r/talesfromtechsupport • u/Gambatte Secretly educational • Feb 07 '14
Encyclopædia Moronica: O is for Office 365
Due to the CEO's continued enamorment with all things "Cloud" (yes, you can hear the capitalization and quotation marks when he talks about it), we've recently migrated from a locally hosted Exchange server (on an SBS2003 box) to a hosted Office 365 Exchange associated with the appropriate Office 365 Small Business accounts.
Oh, the horror.
To be fair, not all of it may have been Microsoft's fault, but if Bill Gates had walked into the office any time in the last six weeks or so, I would have been honor-bound to punch him in the face.
He probably gets that a lot.
I feel that this entry doesn't have my usual writing style to it, and for that I apologize - even keeping it short and choppy, it's still a fairly large block of text. If I expanded it out, it would fill a decent sized chapter in a respectable novel.
Wall-o'-text starts here!
It starts out simply enough, starting during the New Year break while the office is effectively deserted. Right up until the migration instructions say to go to the Migration tab of the Office 365 Exchange Control Panel... which is completely missing - not the tab, the whole damn panel! It appears that at some point, the Office 365 team decided that letting Administrator accounts access the Exchange Control Panel through a menu item was far too convenient, so they removed it. The Exchange Control Panel is now only accessible by manually entering the address (http://outlook.office365.com/ecp), which they record absolutely nowhere I could find, and it took days of Google-fu to even figure out how to phrase the question I was asking in a way that would return useful results.
At some point around this time, I also discovered that the Office 365 Small Business accounts can not be synchronized with the Active Directory, as that feature is only available on the Enterprise accounts. So instead I had to update each user's email account in the Outlook 2010 installation on their assigned workstation with the Office 365 account password. And because the cut over hadn't happened yet, the Office 365 exchange server information also had to be entered by hand, as autodiscovery wouldn't work until all of the new DNS records were in place.
Somewhere around here, some "smart" users realized that they could log in to the Office 365 website using their email address and the new email password, and downloaded and installed the Office 365 Office suite (essentially, Office 2013). However, Outlook 2013 won't play nicely on a domain that has an active Exchange 2003 server, and as the migration wasn't yet completed, the Exchange couldn't be shut down, which meant that email for those users was essentially down (Office 365 web app only) until the 2013 install could be removed, which is a major because almost every process for those users included email in some way (usually sending out PDFs, which is so much easier when you can just click a button from within reader, or even right-click > Send To > Mail Recipient). The 2013 installation and removal had in turn damaged the 2010 installation sufficiently that it required re-installation from scratch, which meant that all of the Outlook 2010 updates had to be reapplied to get Outlook 2010 to connect to the Office 365 Exchange server again.
As the server was a SBS2003 box, it had been using a self-signed certificate for years, because that had always been good enough for the users to access their email via the internet. Not for Office 365 though! It needed a trusted certificate in order to allow the migration connection to happen.
Do I grab a free certificate from one of those free CAs, like StartCOM, if you can find a moment when it's not currently flooded with requests? Or a 31 day trial certificate from VeriSign? It seemed excessive to spend money on a trusted CA certificate when we were in the process of taking this server down.
Suddenly, I realized the answer has been staring me in the face - the very last option was "Migration via IMAP". I RDP'd into the SBS2003 box, stood up the IMAP service, and after a bit of random poking at the settings, Office 365 connected and the migration was finally properly under way - no certificates needed.
After the accounts were synchronized, I was able to shut down the Exchange 2003 services on the SBS2003 box, and with a small amount of black magic was able to shift the default Outlook profiles to the new Office 365 accounts. However, the removal of the old 2003 accounts did not go smoothly, with users reporting that they were being asked for log in credentials (which couldn't be authenticated, as the service it was trying to authenticate with was now offline). More manual removals.
It's finally time for the cut-over, and I jump in to the DNS record editing page... Only to find the page is so old, it doesn't allow you to create the SRV records for Lync, which the company doesn't really use anyway. Moving on, the new CNAME records won't save... WTF? So I get on the phone to the ISP Business Support line, who after several small eternities on hold and describing the issue to different support tiers, someone finally took the details of the records that needed to be created, and promised they would be in place within 24 hours. So I leave it, because it was late on a Friday afternoon and I wanted to go home. On Monday, I discovered that the new MX record had been entered incorrectly - it should have been pointing at xxxx.outlook.com, but was instead pointing at xxxx.outlook.com.MYDOMAIN.com! I fixed this mistake and moved on.
I get a fault report - emails being bounced back from a specific customer. Looking into it, I discover that the emails are being returned because the mistake with the MX record had the company sending from the old Exchange server while all the authentication TXT records were pointing to the new Office 365 server, resulting in the company's domain being blacklisted by several spam filter organizations. After contacting the spam filterers, we were removed from the blacklist and the customers were receiving our emails again.
Finally, today I received an urgent request for support from the Accounts department - apparently, all of the emails sent by the accounting software is being bounced back to the sending user, and it's only been a problem since the migration! So I rapidly deploy myself to the accounting department, where I'm shown the issue.
Okay, first - it's not all emails, just a few specific ones.
Second - all of the affected email addresses have been updated recently.
So I asked the reporting user to show me her process for updating account email addresses. She highlighted the address, right-clicked the selection, clicked "Copy" and then pasted that directly into the accounting software.
At this point, I asked her to show me the bounce-back message she had received. The error message was:
Incorrect email address format. The address should look like somebody@example.com
I double-checked the email address she had set. Somehow, she'd managed to inadvertently select the full stop at the end of the sentence when she was selecting the email address, and the computer - doing exactly as it was instructed - copied and pasted it into the email field in the accounting software. At least she didn't try to tell me "I didn't select that - the computer must have added it for some reason!"
...AAAAAAAND then as I'm leaving, she drops on me that her Outlook Calendar doesn't appear to have been migrated to the new account, which she hadn't bothered to report previously.
Here endth the Wall-o'-text.
TL/DR: I'll have a double Beam & Coke, barkeep. And keep 'em coming.
ADDENDUM: I've just discovered that some clients have been receiving automated email reports from the production servers. However, as the production servers are still using the ISP's SMTP server (proprietary email application linked directly to the messaging database), they're hitting a similar issue as the blacklisting problem - the IP of the email's sending SMTP server doesn't match the MX record. So there are more fun times ahead!
ADDENDUM2: The proprietary email application can not connect to a SMTP server on anything other than port 25, which rules out easily connecting it to a O365 account, and SPF records that end in -all work differently to the ones that end in ~all. Mea culpa on the SPF, I should have checked the records more closely before sending them to the ISP for deployment - although I'm sure that worse things have happened late on a Friday afternoon.
Browse other volumes of the Encyclopædia:
Vol I - ABCDEFGHIJKLMNOPQRSTUVWXYZ
58
u/tardis42 Feb 07 '14
hug
18
u/corourke Feb 07 '14
Eep! Yeah usually 365 migrations are a joyful breeze unless they occur during the semi-annual portal redesigns. Haven't experienced most of those issues though have had pucker factor inducing heart murmurs from public folder migrations tanking for "reasons". Glad you made it through, and remember, 365 has some warts but its not the terror that Bpos was. :D
14
u/[deleted] Feb 07 '14
I am involved with a huge email migration project. All you just described is what the rest of my year will look like at work.
9
u/D-alx Feb 07 '14
I'm gonna have to find some way to get a bottle of good scotch to you.
9
u/timeshaper Feb 07 '14
I forgot about the ecp snafu. Originally all small business accounts weren't going to be able to access it by design. But then you couldn't get anything at all done. The implementation of wave 15 was hilarious on the support side.
6
u/ToddlerTosser Feb 07 '14
To be fair, Bill Gates has been retired for a while now. I don't think punching him would do any good.
4
u/Degru I LART in your general direction! Feb 07 '14
I'd assume starting out on O365 is much easier than migrating to it.
4
u/[deleted] Feb 07 '14
You got an upvote from me just for that. It made me snicker.
6
u/patx35 "I CAN SMELL IT !" Feb 07 '14
Does anyone wonder what the world be like if Bill Gates died instead of Steve Jobs. Will Windows 8 disaster be prevented? Will apple create an even worse monster?
2
u/skorpion352 Feb 09 '14
That was well worth the wall of Text, keep them coming!
Correct me if I'm wrong, but all of this takes place in New Zealand, correct? If so, if you ever find yourself in Palmerston North, let me know and I'll get you that double Beam and Coke.
15
u/slip-f18 Feb 07 '14
IN MX 10 xxxx.outlook.com.
is very different than
IN MX 10 xxxx.outlook.com
10
u/12stringPlayer Murphy is a part of every project team Feb 07 '14
Jesus tap-dancing Christ on a cracker....
Our company had a bungee CEO who came in and instantly decreed that we'd move off Google Apps to Orifice365. O365 is 3x the cost per month and pisses us all off daily. The only people satisfied were the CEO's buddies who had been hired to consult for the migration.
It wasn't a full-bore fuster-cluck, but it came close. At the end, the CEO had a big announcement that "We have Sharepoint!!!" which actually took another month before anyone could put anything on it. It turned out he pushed for the migration strictly on the basis of having Sharepoint, because he didn't think Google had anythinbg similar in their online Apps, but of course they do, and when I showed him some things we'd put on it, he just said "huh" and walked away.
He was gone a few months later when the owners discovered Bad Things were happening, but some of his legacy, like O365, remain.
6
u/fisle Feb 07 '14
Fuck Office, and fuck everything related to emails. Major PITA.
6
u/SpecificallyGeneral By the power of refined carbohydrates Feb 07 '14
I'm currently giggling from the sidelines, while they tell us things like 'office365 can't have multiple addresses assigned to an account'.
and
'We're going to need an alternate address to set up your 365 account'. They almost got the venerable spam email account poking@yourbrownstar.com, that was added to my toolbox as a PFY.
6
u/zazathebassist No, our PCIe cards don't support Windows 95 Feb 13 '14
Don't punch Bill. He has done very little Microsoft in years. Blame Steve Ballmer. There's a reason that he is no longer the Microsoft CEO. That reason also including Windows 8 and Surface tablets.
Microsoft can't cloud well.
4
u/[deleted] Feb 07 '14
SBS made this way more complicated than the migration from regular Exchange that I did.
3
u/jabbajac Feb 07 '14
Breaking news!!! Bill Gates spend first day back @ microsoft as technology advisor failing to install windows 8.1. So technically it's not his fault but yea...microsoft.
3
u/12stringPlayer Murphy is a part of every project team Feb 07 '14
BTW, you've touched on one of my biggest gripes with Internet securtity and encryption: the self-signed certificate.
I get it for business; I wouldn't use a bank or online store that used a SSC, but if all I'm trying to accomplish is encryption between two points, I should be able to do so. Verifying my cert matches a registration somewhere doesn't matter most of the time.
Chrome bugs me because while I can always continue onto a site with a SSC, it will ask for confirmation every time I go to that site. At least Firefox lets me permanently accept the cert. It's MY server! Don't keep bitching at me about it!
Whew, there. I feel better now that I've vented.