This is Part II of a two-part series on how we got started using Azure. Part I focuses on the technical bits. This one is focused on something else: Business, Limit & billing.
Here begins the non-technical story of where our successful tests became a problem. We were using more and more clusters. Mostly we set them up, use them for a day, stop them (not delete) and start them again when we need them again. We would have various clusters sitting there (ELK, CDH, HDP, secure, non-secure etc.) for whatever we need. This was when we first ran into quota limits.
Azure enforces limits on how many cores per instance type per region you may use. The default limits are fairly low (10) and we were able to easily increase those to 200 which allows ~20-25 good nodes. 25 nodes doesn't sound too bad, but this is not enough for having a few clusters around, because unfortunately every cluster (whether started or stopped) counts against your core quota.
So I started the process of increasing the quota again. Last year I already tried to speak to anyone from Microsoft sales to establish a relationship but failed. I tried calling, mailing, tweeting but as we are a small company so we seem to be under their radar. We spent on average maybe 200€/month on Azure at the time. So, for this latest increase request I immediately contacted support via the Portal (which has a lot of features but is so incredibly slow!).
The answer I received was a request for lots of financial information:
Please provide us your latest 2 years of Audited Financial Statement as we need to conduct a financial analysis as your requested credit line exceeded our maximum auto approved threshold.
We're a startup that didn't even exist for two years back then. Also, we don't need to be audited and we're based in Germany and this seemed to be very US centric. And last but not least I'm not a finance expert so I asked to close the case for now. In the meantime, my colleague Sönke started writing Terraform scripts for AWS which are ~70% done but we haven't had time to finish them.
A few weeks ago we were struggling with our core limit again so I picked up my conversation with MS support. This time I insisted on a German speaker in the hopes that the bureaucracy (while unavoidable) would at least be in German and geared towards the German market and rules. I was wrong. I picked a random high number of cores (2000) as we didn't have any specific hard requirements now.
A very nice lady contacted me and we've been in contact ever since and she's helped me through the process. She needed some information for the Accounts Payable contact etc. (we're small, just four people so that'd be me...) and a bunch of other things and then she again forwarded me to the US credit risk department where things started to be in English again.
It turns out that we have a core limit that's based on our credit limit. This core limit assumes that I'm going to use these cores 24/7. An example: 1 Core costs 1€/hour. So, it exposes Microsoft to a credit risk of 1€ 24h 31 days = 744€. Our request for 2000 cores would therefore expose us to much more: 172,800 USD. True, that's probably how much money we could spend but it's not our use-case at all (more on that earlier and later). Unfortunately, that's the way things work. The maximum that we could get without any further checks was 50.000 USD which is 578 cores (better than nothing, we'll take it).
But I didn't want to stop there. I always try to be proactive. If there's a process that I have a high chance of needing sometime in the future I'd rather start it now and learn all about it now while I still have the time rather than doing it in a hurry.
So I learned that in order for a credit increase above 50.000 USD we needed to do two things:
In addition we needed the following:
To proceed with your request, kindly provide us the latest 2 years of Audited Financial Statement of your company. The statement should include a balance sheet, income statement and cash flow to proceed on conducting financial analysis as your requested credit line exceeded our maximum auto approved threshold. We will assess the financial stability, liquidity and profitability of your company.
Fortunately we've got an accountant and I forwarded it to him but even he can't change time and regulations so we're still not able to provide two years' worth of data and none of it will be audited.
For the age of the company we have a solution:
Or, since this company is newly founded (2016), we can suggest to undergo securitization. But that is only if you also switch payment from credit card to invoice PI as we cannot offer securitization for customers under credit card mode of payment.
I've never heard of "securitization" before, so I asked. I also asked how other German startups deal with this situation:
Most of the time, German start up companies accepts to be granted with the maximum cores that is equivalent to their market threshold. (578 cores which is equivalent to 49,939.2 USD). But there are others who choose also to provide securities.
Regarding securitization, the type of security (security deposit or SBLC) and the amount to be paid for the security will be based on your total credit exposure. This will also be the amount to be set as your credit limit.
What we can suggest for now is since you have mentioned that the 2000 cores was picked randomly, we can grant you the maximum cores of 578. But if you really need higher than 578, we can also suggest to switch to invoice mode of payment and undergo securitization (again, this will be based on your request).
Okay. So securitization seems to mean that we provide Microsoft with some form of security that they'll get their money. I obviously don't want to put in a security deposit of tens of thousands of USD for an expected monthly invoice of maybe 200-500€. So I contacted them again:
As for the securitization process, if your equated credit exposure (cores*86.40) is less than or equal to 100K USD, you will have to provide a Security Deposit. If it is beyond 100K USD, you will have to provide a Standby Letter of Credit. We cannot proceed to discuss further until the below has been confirmed:
As you can see above we were granted an increase to 578 cores which I then requested. Unfortunately, it was subsequently denied because there were not enough resources available in our region for any more cores for Dv2/DSv2, Dv3/DSv3 and Ev3/ESv3 machines. So, the whole process has been for nothing (apart from learning new processes).
This is not the cloud experience I signed up for :)
Fortunately a few weeks later we were finally granted our 578 cores. That should help us for now but has made us wary.
The cloud promises unlimited choices, flexibility and especially elasticity. This might be true on paper and it will probably be true for some (or even most?) companies but for us it definitely is not true and I'm fairly certain that we can't be alone.
I understand Microsoft and their need to protect themselves with credit checks. But our first contact here was immediately the Credit Risk department in the US. I would expect a sales person to reach out to us and I would also expect the whole process to be better documented laid out and internationalized for a company like Microsoft. We cannot be the first company that doesn't know what a Standby Letter of Credit is. Even our bank had to take three days to figure out how it works. Those things are just not normal for the German market.
Our use case might be somewhat special (having lots of stopped machines around) so while I'd like to have it different I kind of understand that Microsoft is the way it is in terms of credit checks. We'd be perfectly fine with a credit limit of say 5.000€/month and then have our machines stop once that is hit. Or do not count stopped machines against the quota. Either way: This is not the flexibility and elasticity I had hoped for.
So if you're a new company/startup and want to move to the cloud think about these things early!
We did reach out to Google Cloud and Amazon AWS as well as Hetzner Cloud for comment.
Google did not respond at all. We filled out their contact form requesting a sales person to contact us. It's been month and we haven't even had an E-Mail. Me mentioning our expected monthly spending must have turned them off?
Amazon responded within a few days and answered all my questions as best as they could. They were very happy to help but not very open about their processes. They would not reveal which kind of checks they do when requesting an increase for cores. But they did assure me that they'd rather approve than reject requests and that those limits are mostly in place to protect the customer. They also couldn't help but chuckle at the lack of resources in Azure and promised us that this wouldn't be a problem in AWS.
Hetzner also did not respond with details about their process but I got the feeling that they might not yet have as detailed processes as the others and they seemed happy to increase my limits. For them the limits are a bit weird as they seem to be based on the number of machines no matter the size of those machines.
I haven't contacted any other cloud providers but would be happy to if you're interested.