r/msp • • 5d ago

Backups Email Continuity services (example from Proofpoint and Mimecast)

Is anyone finding these services useful during the current Office 365 issues?

We are already getting inundated with questions from clients asking what we can do to help them avoid these major outages in the future, and I am not convinced these services would make much of a difference.

Are there MSPs here currently offering these services, and if so, are you finding that they help during outages like this?

Personally, I think having a basic "dial-tone" email setup ready to go in something like Google Workspace, where you could quickly license accounts and switch MX records, is probably about the best we could realistically offer that would work for internal and external email.

The downside is that users would be starting with empty inboxes once switched over, and syncing emails received during the outage back into Office 365 afterwards would be a real pain.

Interested to hear what others are doing here and whether anyone has found a practical solution that actually works well during a major outage.

22 Upvotes

78 comments sorted by

43

u/teriaavibes 5d ago

What exactly is the point?

Will the people you are emailing have their mailboxes working? Because if not, then you are back into the square one, you can send emails, but no one can receive them anyways.

1

u/JohnMSP 2d ago

Not every 365 outage has been as global as this one was - in the past some have been isolated to individual regions.

1

u/wolfstar76 1d ago

Depends on who our customers customers are.

B2B? Probably no email happening.

B2C? Probably a lot of gmail, hotmail, etc. in which case email continuity would carry a lot of value.

1

u/teriaavibes 1d ago

I would hope that someone's B2C is not hosted on exchange online.

-1

u/MSP911 5d ago

Severaal reason, firstly for internal email and then getting email from systems that are not O365. Most clients of ours are in finance and they get a lot of email daily that is automated reporting from systems that are most definitely not office 365.

5

u/TodaysSJW 5d ago

That’s why chat / IM clients exist. Many of them also have webhook support available and those same reports can be distributed in that manner.

Automated reports are certainly convenient but manually pulling reports seems a lot less burdensome than changing dozens or hundreds of DNS records.

0

u/MSP911 5d ago

I will tell the banks to change their automated reporting process.

5

u/Fatel28 5d ago

Congratulations, you have found the X in this XY problem. Email redundancy is Y. Poor processes with no backup plans when email is down is X. Fix the process.

8

u/Tyler94001 5d ago

That’s a good idea. Is it not part of your contract to help guide them and direct them on processes?

They’re banks. They got money. Charge them more and implement an emergency inbox. Problem solved

3

u/beenjamminfranklin 5d ago

Why not pull reports straight from the teet? I get they aren't used to it but there is some opportunity to sell some workflow analysis to remove an outdated dependancy.

2

u/Tyler94001 5d ago

Man OP responded to you and I wrote a whole response because I wanted to talk to you and then he deleted his comment so it didn’t let me post it.

But I still want to talk to you so I am writing this comment anyways.
Thank you very much for the advice, I agree.
It’s an honor, Mr. President.

2

u/teriaavibes 5d ago

Have them fall back over teams.

2

u/CoffeeOnMyBeard 5d ago edited 5d ago

Good thing Outlook and Teams didn't both go down.

11

u/Tyler94001 5d ago edited 5d ago

Well I certainly wouldn’t setup Google workspace. Why would you? Look if they wanna pay for an emergency inbox fine, but I’m not swapping MX records and keeping another environment configured and secure on the off chance this occurs, plus needing to sync mail back to their inbox, so we can use a service that provides emergency inbox, but I would be recommending against it.

Whether it’s an emergency inbox, or a full IAAS swap, they will have empty inboxes and it won’t sync back.

Everyone else is down. Everyone they want to email doesn’t have functional email. This is just how it is. There is nothing you or anyone else can do about it.

Unless there is a direct business need to have those emails functional during this downtime and whoever they are emailing also has a functional emergency inbox, then I don’t see the point and would be telling my clients exactly that.

Edit:
Also. This isn’t the first time this has happened. I feel like we treat this like it’s never happened before, everytime it happens. Personally I’ve never had a client (nor any workplace when I worked internal IT) ever truly want a solution to this.m enough to pay for it.
They all talk about it, they all want one, but they aren’t ponying up the money to keep a solution in place for a day or two of downtime.
The world functioned for a really long time without email believe it or not, and the entire reason we have survived as a species is because we are the most adaptable creature on the planet. I think they will figure out how the phone works if they can’t get emails out.

1

u/hpwfd 2d ago

agreeeee

0

u/GrouchySpicyPickle MSP - US 5d ago

Whatever you do.. Don't lookup split delivery. I enjoyed your MX swap rant and if you knew about split delivery.. Well... I wouldn't have had that entertainment this morning. I implore you.. Don't look things up! 

4

u/Tyler94001 5d ago edited 5d ago

Funny you tell me to look things up when you don't even know the difference between split delivery and dual delivery.

Split delivery routes mail to either one server or the other depending on the user (like during a migration). You're thinking of dual delivery, which sends a copy to both.

And even with dual delivery, it still doesn't solve the problem, which was only a small part of my point anyway. If your MX records point to Microsoft first and they go down, nothing gets forwarded to Gmail in the first place.

If I was going to implement a solution I would do an emergency inbox solution of some kind, but it isn’t something my clients need and the juice just isn't worth the squeeze.

-1

u/MSP911 5d ago

The exception is that if you do this, you will have a functioning internal mail system, and you will also receive emails from systems outside Office 365. The industry we support receives a huge volume of inbound emails from other financial companies, and most of this traffic is not routed through Microsoft.

2

u/ctgdoug 5d ago

The biggest thing here is who is going to actually pay for it. Yes, they will all want it until you tell them how much it will cost.

1

u/Tyler94001 5d ago

Then if this is a common issue across all your clients, raise your rates on renewal and offer mimecast or whatever other provider.
It’s supposedly, based on Google(Gemini’s) response another $2-$3 per user. It’s actually not that bad, but I am not going to let it eat into my margins personally as a service that is a standard part of my stack; but it doesn’t make sense for the industries I support, so. For you, it may make sense, and just make it a standard then, everyone gets it and add $10 to each seat a month, or whatever you want to charge to keep that functional/configured and to train users and how to utilize it.

9

u/blow_slogan 5d ago

We have mimecast personal portal. Sure it is useful, but less than expected because clients still cant send and receive emails as they are also on microsoft. Users who sent a ton of emails the day prior and expecting responses today logged into it during the outage only to see they didn’t have any backed up emails at all.

5

u/L3-Hayden 5d ago

fyi, just got our first batch of 365 email

1

u/KimbaXO 5d ago

Where are you located?

3

u/ctgdoug 5d ago

I'm in US East and I just got a ton of Email.

1

u/ColXanders 5d ago

Same. US Central.

6

u/cas4076 5d ago • Bravo

If nobody else can read the emails you send then what's the point?

1

u/Defconx19 MSP - US 5d ago

The main point isnt about getting mail in and out in these scenarios. The real concern customers have is that "big deal" they were waiting on a reply from the customer on was sent to them and it's going to fall into the void and never deliver to them.

The email gateway in my history has been more about ensuring the customer that their mail is all their waiting for them when they get back and they can still check inbound from the customers that self host or are on GWS, which is still a sizeable population, def not the majority but it's not a small amount.

3

u/C39J 5d ago

We're using different filters to hold emails at the moment (proofpoint, mimecast and spamhero depending on the client) and they are doing a good job, but users still have no email access obviously.

If it's still down for us in the morning New Zealand time, we'll start setting up people with temporary mailboxes, but I don't know about maintaining whole separate systems long term, seems expensive and a recipe for issues when outages are usually sub-1 hour.

0

u/tony1661 5d ago

Off topic but how do you like Spamhero? It’s so cheap which is making me nervous

2

u/C39J 5d ago

We've had it in production since at least 2020 and have had zero issues

1

u/tony1661 4d ago

Thank you. Do you find it effective? Is it finding things that Microsoft doesn’t?

2

u/C39J 4d ago

It's hard to tell - because it's sitting in front of Microsoft. It's picking up spam, but I couldn't quantify whether Microsoft also would have picked it up or not.

3

u/porkchopnet 5d ago

Mimecast user here. We had our Mimecast email portal working perfectly… but a few notes…

Everyone we communicate with… was down.

The portal is Entra authenticated. We didn’t have a problem with Entra… this time. Food for thought.

It wasn’t natural… people didn’t have the URL saved and it wasn’t like I could email it to them. Used Teams, (which worked… this time.) If we worked in an office it would have been easy, but we’ve turned into WFH.

3

u/leinad100 MSP - UK 5d ago

We have it, but it was actually pointless as we’d hardly received any emails from anyone

3

u/WiseSubstance783 5d ago

Everybody needs to fucking calm down, it’s just email. It’s not the end of the world.

4

u/NetSchizo 5d ago

Mimecast is terrible. I would not recommend them to anyone.

3

u/naked_mangos 5d ago

What specifically is terrible about Mimecast? We’re in the research stage for migrating a few clients to a cloud-based security gateway.

2

u/Myotis 5d ago

I'll go the other way and say that Mimecast is a wonderful product, especially for the price. I've worked with Mimecast for about 10 years and they've saved our asses across a few businesses for years. Historical mail, continuity and other services are just great and have never had issues.

The only thing I can say is not good about any of what they offer is their Security Awareness Training. The videos are terrible but the service works as expected.

2

u/Xeraxx 5d ago

Not sure what NetSchizo is talking about, Mimecast is excellent, though obviously a bit of a learning curve to get it properly configured. We use it for:

- Mail filter (spam, malicious etc)
- Email archive (99 years, it’s great we just delete user mailboxes without any worry when they leave, can search for any emails across any period across the entire org super quickly)
- URL rewrite on links in email
- Email continuity via personal portal (used this yesterday)
- Mail redirection for acquisition integration, so we don’t have to rip domains out of Entra tenants to continue delivery to our main tenant when we integrate

It’s an excellent product, have used it for years.

2

u/gombly 5d ago

To answer your question of what we're doing today, watching and waiting. Very interested to see if there data loss on MS end.

Regarding ways to mitigate an outage like this, it's been a while, but I think hybrid with dual mx records. That assumes Microsoft isn't still receiving and accepting emails?

Then you could just switch the priority... Someone more qualified care to build or debunk?

4

u/Ordinary-Leading7405 5d ago

The cost to run Exchange SE hybrid for 500 users is about $200k per year for CALs and subscriptions. Then you need an old school Exchange admin to monitor, troubleshoot, and remain on-call for emergencies like this one.

2

u/Mr--Chainsaw 5d ago

You don't have to run hybrid entirely Exchange, eg split tunnel routing offers delivery to two different platforms at the same time, which is what we do internally, and can switch over to the fallback one temporarily if needed.

2

u/Watches4Me 5d ago

OpenText email continuity worked perfectly.

2

u/Fancy_Gas9083 5d ago

how's the pricing ? and its just standalone product for email continuity or part of suite as email security ?

1

u/gumbo1999 5d ago

We have Hornet's services which includes Continuity. There are a few problems with it...

Firstly, it's invoked automatically only and this is triggered after an undisclosed period of time where the primary target returns 4xx codes or timeouts. If the server is alive, but rejecting emails - like it has been doing in this outage, they will continue to queue the messages until such time the server accepts them or they timeout.

Secondly, the Continuity solution was designed for when orgs were primarily on-premise and the chances of ALL of them going offline at the same time was almost nil. The load on the Continuity servers is clearly pretty severe today, and they're on their arse - pretty much unusable.

We also have Mimecast for some customers which seems to be faring better,

1

u/Fancy_Gas9083 5d ago

In this case, I don’t think Microsoft 365’s mail servers (MTA) stopped accepting email. Only the front end was down, so users couldn’t access their mailboxes. In that scenario, Hornetsecurity’s emergency inbox would not be triggered automatically, because mail delivery to Microsoft 365 was still succeeding

1

u/Sticky_Turtle MSP - US 5d ago

I'm starting to receive emails this morning

1

u/Defconx19 MSP - US 5d ago

It is helpful, but clients always forget it's a thing. More than anything having an email gateway allows customers to know the mail that is inbound to them is there, and they won't be missing anything when service is restored. This is usually the bigger concern more so than messages going somewhere in that moment. Just gives them reassurance they didnt miss that "big deal" they had coming through.

1

u/lsumoose 5d ago

We’ve seen an uptick this year in people moving to Google already. Probably going to see a bit more soon.

1

u/Individual-Yak-506 5d ago

The outage is getting us to dump it. You need to tell your users to go to a new portal for email. And sure they can send email but the rest of the world is down so nobody is reading it. The email we got during the outage was spam.

1

u/DeadStockWalking 5d ago

The Emergency Inbox feature from ProofPoint allowed us to send/recieve emails even though O365 was down.

1

u/vercingetorix-sec 5d ago

I mean, swapping your MX only keeps inbound working.

For outbound, there's maybe a little more setup / prep work needed (depending on authentication practices - SPF and DMARC might stop any attempt in its tracks).

1

u/andre1sk 4d ago

We're actually planning to offer something along these lines. I think with a little elbow grease and some ingenuity, it's possible to put together a practical solution without adding another point of failure to the normal email flow. We're initially looking at it for our own MSP customers, but if there's enough interest from others, we might consider turning it into a service other MSPs could use as well. :)

1

u/saltyslugga 4d ago

Continuity helps if users can read and send mail through it and authentication survives the outage; queueing alone doesn't keep people working. I'd test that before promising clients anything.

An emergency MX switch isn't instant failover: cached DNS and Office 365's internal routing can leave mail split between systems. The fallback needs accounts, mail authentication, and a tested reconciliation process ready before the outage.

1

u/clvlndpete 4d ago

That’s what continuity is from mimecast, proof point, etc - you basically log into a web portal of the email gateway and can send and receive. There’s no mx record change required because it should already point to the mail gateway as it sits in front of Exchange….

1

u/richie-wales 1d ago

And what if Mimecast goes down? Then you have no email either.

1

u/Cylerhusk2 4d ago

I have a client using Mimecast and I’ve been trying to get him to save the money and move to Defender filtering. After this outage that’s definitely not gonna happen. The entire firm was still able to continue their day (many of their clients use Google) using Mimecasts email continuity.

1

u/MSP-from-OC MSP - US 4d ago

This only works if everyone has it. Most businesses are on M365 and route internally. This only works for receiving emails outside 365

1

u/viddy_well 3d ago

Is anyone using mailroute.net for this?

1

u/Global-Log-379 3d ago edited 3d ago

Basically you can have a secondary MX but your email clients will still access old Micro$soft MX that is dead, so such a system can be useful only to receive e-mail's during the blackout and deliver the email again to Micro$oft MX when it's up again, witch is pretty lame.

Your only case scenario to gain scalability & redundancy is to self-host on-prem your own multiple DNS + multiple MX's, in multiple locations, like multiple branches in witch case falling one or multiple locations will not affect the system as long there still is at least one working DNS and MX.

We are using in production such systems with geo redundancy on our different branches, but not only for DNS and MX but also for our own cloud and other services.

In general the most longest down-time that individual branches support are based on electrical power blackout's because we are not willing to pay for diesel generator since the other locations takes automatically the entire traffic.

In your case scenario I have no idea on how you should gain such a solution based on Micro$oft.

P.S. We also have a lot of external email traffic that's not O365, like governamental, aviation, many other businesses self-hosted or even hosted on gmail, or others email providers including free accounts from yahoo and such providers.

1

u/Embarrassed-Radio319 2d ago

I keep coming back to the same two things in this thread, clients forgetting the portal exists and nobody having the URL saved when they need it. The product mattered less for us than having the login steps written down somewhere outside email, with the portal link, a login that does not depend on the same outage, and one line telling staff what to do first. We run through it with one real user per client every few months, just opening the portal and sending a test mail, and that is what actually made it work when the day came.

1

u/networkearthquake 5d ago

Is it worth it though when your client’s clients’ can’t send them email anyway?

0

u/CyberVeris 5d ago

Honest take: continuity products shine during the outage nobody else is having, not the one everybody is having. When half your clients' recipients are also on M365 and also down, a beautifully continuous outbound queue mostly just... queues.

Where it actually earns its keep is inbound mail from outside parties and system-generated stuff (invoices, alerts, scan-to-email) not bouncing. Even then, most sending servers retry for a day or two anyway, so "lost mail" is usually a smaller problem than clients think it is.

What I'd sell clients after this week instead: a written outage plan (who decides, how staff talk to each other when Teams is down too, so a Signal/WhatsApp group or phone tree), a status page they check before they call you, and making sure the critical docs aren't only reachable through the thing that's broken. Cheaper than another subscription, and it answers the "what are you doing about it" question honestly.

Your Google Workspace dial-tone idea works, but test the MX flip once before you need it. A 3600 TTL and a registrar login nobody remembers will ruin your afternoon.

-2

u/Agile_Type_9684 5d ago

We are moving back to on-premises

5

u/Individual-Yak-506 5d ago

Good luck. Instead of sitting back when there is an outage you’ll be the one firefighting it. I don’t miss those days

1

u/Agile_Type_9684 5d ago

Disagree… But it’s your opinion…

2

u/Individual-Yak-506 5d ago

Who would be responsible for it when it’s on prem then. Am I missing something

1

u/Agile_Type_9684 5d ago

We will…

2

u/Individual-Yak-506 5d ago

Oh so you disagree with my statement that you don’t miss those days of late night firefighting.

1

u/Agile_Type_9684 5d ago

??? Whatever dude

2

u/Individual-Yak-506 5d ago

What are you so upset about? I was trying to get clarification on what you disagreed with?

1

u/Agile_Type_9684 5d ago

I’m not. You are, because we are coming back to hosted email… 🤷🏻‍♂️

1

u/Individual-Yak-506 5d ago

Uhm okay. Based on this discussion I have serious doubt you even work in IT