Too Much Automation
-
I've seen automation complexity create a lot of minor outages where having a human do the work would have been easier and safer.
-
Automation so rarely survives the person who put it in too. It's a tough situation, because there are very real benefits to it.
-
Automation in small doses is good, but on a large scale can definitely add unnecessary complexity to a site, unless you have some checks & balances. Also, if you automate things too much, why do they need people working?
-
Automation is the inevitable trend. Microsoft with Azure, Amazon with AWS, and Facebook have smart folks only able to continue doing business by automation. LISA has great talks on it (okay, boring to non-I.T. folks), at times by these people.
There is a reason Google "sysadmins" are called SREs (site reliability engineers). Google has a process for he automation, documentation, change control, and following up on the inexplicable weirdnesses. Calling them "engineers" is apt, as they need to go to a low level to get an understanding of issues, and craft appropriate checks & workarounds.
-
@RoguePacket said:
Automation is the inevitable trend. Microsoft with Azure, Amazon with AWS, and Facebook have smart folks only able to continue doing business by automation. LISA has great talks on it (okay, boring to non-I.T. folks), at times by these people.
They also have one thing in common.... only running one app across their entire estate. They are only automating within a tiny scope.
-
Google, Amazon, and Microsoft have one app? Can see each them having their own "single" distributed OS with myriad products running underneath it.
Each has had issues, emphasizing automation isn't perfect and remains a challenge.
-
@RoguePacket said:
Google, Amazon, and Microsoft have one app? Can see each them having their own "single" distributed OS with myriad products running underneath it.
Each has had issues, emphasizing automation isn't perfect and remains a challenge.
Yes, one app. THEY provide one thing, a vanilla platform. That's why they can automate. All of the complexity is on top of them. THEY only deliver a very simple, very singular thing. Or a very few things, on huge scale. They don't have a different product on every server. They have tens of thousands of identical servers with no identity of their own. To them, it's one big computer. That's why automation works so well there.
-
Automation is great, but I think an important rule to remember is "Code to the exception." I was taught this in programming (in my "previous life" as a programmer) and I rarely had to go back and fix code for an module or app, unless it was for new functionality. While you can't catch everything, it does force you to search out possible failures and/or errors that might happen. I've also heard of it referred to as "What If?" coding.
-
@scottalanmiller said:
Yes, one app. THEY provide one thing, a vanilla platform....
Given these are proprietary systems, information is scarce. Might you know a source describing their vanilla platform? Tend to think in terms of what OpenStack is presenting.
-
@RoguePacket said:
@scottalanmiller said:
Yes, one app. THEY provide one thing, a vanilla platform....
Given these are proprietary systems, information is scarce. Might you know a source describing their vanilla platform? Tend to think in terms of what OpenStack is presenting.
Exactly. Amazon just provides EC2. It is functionally equivalent to OpenStack. Amazon doesn't provide all the stuff on top. Everything is just hundreds of thousands of identical images under a single interface. That's what an IaaS cloud is. Azure, same thing. It's one, extremely basic thing replicated to a crazy degree. Nothing like what an IT department runs. In IT we run a different app on every server. You can't just make blind copies of your database server or your domain controller or your file server. Each machine has a specific role and does something different. Amazon isn't like that. To them, every instance is absolutely identical and provides nothing but an incredibly bare operating system - and they don't even automate the OS, just the layer right beneath it.
-
There is no scarcity of information about them providing a replicated, vanilla platform.
