So I had a visit with Alex Ao

Last Friday (seven days ago) I sat down with Alex Ao, one of the founders, and currently the VP of Business Development at Innosilicon.

Here’s his business card that he gave me (phone numbers blacked out).

A few days earlier Innosilicon had announced a Zcash hardware mining unit that does 50 KSol/s at 650 Watts, named “ZMaster”. That’s 5 times as much hashes per second as the Bitmain Antminer “Z9-mini”, at about 2 times the power draw. The current list price for Innosilicon ZMaster is $9999 dollars. The Bitmain Z9-mini was already selling at $1520 dollars, and a few days ago Bitmain announced a halving of the price to $760 dollars per unit.

The meeting was brokered by Dovey Wan of Danhua Capital.

Here are my recollections from the meeting, informed by some handwritten notes I took during. I might forget or misremember some bits. By writing this, I’m not asserting that I believe everything Alex told me, nor am I asserting that I disbelieve anything. I’m just reporting what was said.

Innosilicon is an IP (chip) design firm. In 2014–2015 they made a 28 nm BTC miner, and in 2015 they made a 28 nm LTC miner. They started design work on a ZEC miner about four months ago, in February 2018. (I was surprised that it took them only four months from beginning design work to having a large number of live units in production!)

He said they currently have 1000 ZMaster units in production, mining on the Zcash mainnet. They’ve been live for a couple of weeks. These are the units that are for sale—they are intended to be shipped to customers. They are mining on them right now for testing and aging. A first batch, they would mine it for a week and then sell. Later on, subsequent batches, they would just run a couple of days.

I asked what mining pool they use. He said their own private pool and also that they test on different pools.

I asked if I can publish this stuff, he said no problem.

He said that their whole idea an “open ASIC”, that they want to support the ecosystem. He said that they would be happy to sell others the chips and they can build the miners, or that they would be happy to licence the IP and let others fabricate chips. I said that this sounds related to Josh Cincinnati’s recent idea about an open ASIC and they should talk.

He said that they also made a Monero miner. I asked if they ran a Monero mining farm and he said yes. I asked when they started and he said he wasn’t sure because he wasn’t the manager of that project but he thought it was last December. I asked how many units and he said fewer than 1000.

!!! I was excited, because this means we’ve found some of the mysterious Monero stealth ASICs! When Monero switched their proof of work in April, the hash rate fell by about 380 MH/s, and subsequently that much hashrate started mining on Monero forks like XMO and XMC. If Innosilicon was running up to 1000 units of their “A8+ Cryptomaster”, which does 240 KH/s of Cryptonight hashing each, then that accounts for about half to two-thirds of the mystery hash-power!

I asked if they work with Bitmain. He immediately said “No!” and laughed. “We are competitors!”. I didn’t think he was lying.

I asked if Innosilicon worked with ASICMiner. He hadn’t heard of it. I showed him ASICminer Equihash 40K profitability | ASIC Miner Value, and he said he hadn’t seen it before and they don’t work with them.

I said that I thought they were missing out on a lot of sales. I said that a lot of potential customers have been contacting me, asking if the Zcash community was going to change the Proof-of-Work to be incompatible with the current ASICs, and that I always tell them I can’t say — that I don’t know what the community will decide. I said that these customers are reluctant to buy, because if the community changes the Proof-of-Work then the miners will produce a lot less value, like the “XMO/XMC” miners now. He nodded.

I said that I thought if he could get in touch with The Zcash Foundation and get them to make some public commitment about not changing the Proof-of-Work, or about including both Equihash-200,9 and other things in the future Proof-of-Work, then he could get a lot more sales.

I said that what would increase trust, for me personally, as well as in my opinion for many other members of the community, is transparency. I said that what I wanted was a commitment from Innosilicon that they would disclose to the public all shipments of Zcash miners, including what size batches they are selling in, as well as to disclose all deployments that Innosilicon does themselves of running Zcash miners, as well as what policy they have about which blockchains, transactions, pools, and precisely what software they run (i.e. which specific verson of zcashd, or if they run their own patches to zcashd). I explained that this latter part would help the dev team diagnose and respond to issues on the network if we knew more about precisely what software the miners were running.

I showed him the announcement and the tweets from Bitmain disclosing their shipments of the Z9 mini, which shows that they’ve shipped the first batch of 77 units, total. He said if you ship 77 units it is okay to disclose but for 1000 it would be a different story. I said that I understood that, but that what I want is a commitment to disclose all shipments and all deployments and not to omit any shipments or deployments. (Which, by the way, I haven’t seen a clear statement of a commitment to that policy from Bitmain, and I think that would be a great improvement to their transparency initiative.)

Also, I said, come to Zcon0 and meet Josh Cincinnati and everyone. He happily agreed.

[Note: after I wrote this, Josh Cincinnati reminded me that he won’t be physically present at Zcon0 because he’s about to have a baby! But Josh will be cyber-present, and there are plenty of other members of the community that Alex can meet there.]

[Note: after I wrote this but before I posted it, The Zcash Foundation issued an open request to Bitmain, Innosilicon, et al. to commit to transparency with a similar list of requests. This is no coincidence! The Foundation and me have been talking about this.]

He said again that Innosilicon’s background is in IP (chip) design, and their strategy is to promote the whole ecosystem. He said that he had helped negotiate the deal by which Litecoin added Segwit. He said something like:

We made contributions to the Litecoin Foundation and funded developer to make more features. Then a certain party blocked the addition of Segwit on Litecoin [note: he meant Jihan Wu]. Then I convinced everyone to support Charlie to introduce Segwit, which made the price go from [some number that I didn’t write down] to [some other number that I didn’t write down].

He told me more about the process of breaking that deadlock that I didn’t write down, but one detail stuck in my memory: he said that at one point the certain party that was blocking Segwit threatened to use its substantial Litecoin hash power, and he revealed that he had even more Litecoin hash power. This detail stuck with me after the end of the conversation…

The conversation was winding down. I asked again if they intend to run a Zcash mining farm. He answer: that’s not our intention. If the market is not well, if I cannot sell them, I will run them. Secondly, for testing and aging. I cannot play as both athlete and referee.

I asked, why were you running the Monero mining farm? He answered, we had a big customer who wanted to buy the miners, but they don’t have electricity, so we run the farm for our customer.

[Note: after I started writing these notes, Zcash engineer str4d made these graphs of mining pool hashrates.

Maybe Innosilicon’s 1000-ZMaster-unit testing farm is the one labelled t1Q2Q and t1XE1? I’m not sure 1000 ZMaster’s would produce that quite that much hashrate, so perhaps Innosilicon has more than 1000 running, or perhaps someone else is also pointing miners at the same pool? Or maybe that is the right hashrate for 1000-ZMaster’s?]

P.S. After I write this whole thing and just before I posted it, Dovey Wan from Danhua texted me to say that there is another new pool which quickly gained 40% of the current hashrate! I haven’t looked for the relevant stats from str4d’s new visualizer yet. Never a dull moment!

15 Likes

t1XE1 about 200-250 Mh\s, 0.05 Mh\s (ZMaster) * 1000 = 50 Mh\s. Bitmain don’t pre-mine, Innosilicon - 50 Mh\s. So who did it? I think they kid on you

1 Like

Do they manufacture in North America? This pertains directly to their competition with bitmain as they’ll be paying more for various imported electronic components from the US, I don’t know how many if any bitmain uses, just a thought
Heres the list of their hiked imports

Thanks for these interviews, @zooko. I’ve been very critical that the rise of Equihash ASICs caught Zco by surprise, so it’s great to see Zco and the Foundation catching up to speed so proactively.

1 Like

I agree, it just doesn’t fit up and some things are strange:

  • There is no unknown pool that fits even 50 Mh/s only, but only one that fits 200+ MH/s. There are only 2 options possible in my opinion. Either their pool they test and mine is only <3% of Equihash network at the moment, which i somehow doubt or it’s just that big new pool ending with snupka. This would mean 4.000 Units if i calculation is correct.

  • Regarding the previous point, the other suspects for the big new pool could be Bitmain or Asicminer, about Asicminer i’am not sure if they have the capacity to release a sudden 5.000 Units for “testing” as they want to ship in July (at least announced it). The other suspect, Bitmain. possible, not sure if their 40k miner is ready and if they would risk such a big hashrate in secret after the talk they had with Zooko. But just in case, one of the other suspects is that big pool, where are the 50 MH/s from Innosilicon than?

  • I’am not sure if i buy the story with a customer purchasing large amounts of cryptonite Monero miners but having no electricity, i have my doubts this is true.

  • Maybe i’am wrong, but from the wording and reading it looka like the Innosilicon guy isn’t that coorporative but mostly avoids anything related to coorporation, transparency, disclosing… Maybe i’am wrong, but that’s how i read it.

  • About the request for transparency made by the foundation:
    I don’t think it’s written good and i’am missing some pressure on the Asic manufactures in it. In my opinion it shouldn’t be just voluntary, this leaves all doors open to share or not share, not disclose or not disclose, and so on. I’am as well missing some harder words in the direction that IF such private and secret mining continues Zcash will take measures not tolerating it any further, including forks/ tweaks/whatever to enforce the transparency or to leave the asics alone. Seriously, that 40% pool just crossed the border and went to far in my opinion.

Thx for the interview/talk/conversation and letting us again now. Wouldn’t it be easier to record such a conversation than making notes?

3 Likes

Zcash as become officially a joke as principles of crypto-currency is being violated not a good long term investment, it appears to be trending to centralization, I can’t believe coin developers are actively colluding with ASIC manufacturers…the decision as been made for Zcash to go ASIC and it is SAD!..save this post end of year they will reverse themselves as the currency will be on its last legs and dying.

9 Likes

For my taste this goes to far. If you think it’s a scam, just bail out and move on …

Actually with a scam you lose money, i think so far you only made money with Zcash …

2 Likes

@zooko, I really don’t understand your position wrt ASICs.

You’ve just related an account in which two people and their companies engaged in a power struggle over the protocol evolution of a block chain, by wielding their hash rate as a weapon to each try to get their own way. And who had the highest hash power was in fact decisive.

That is centralization. A single individual’s opinion won out regardless of any technical argument for or against the change in question, or its effect on Litecoin users, because that individual controlled the majority hash rate on the chain – something that was irrelevant to the merits of the change. [*] This is precisely one of the things the ASIC-skeptical camp has been warning you about.

It doesn’t matter which specific side won in this case, or what the merits of Segwit are/were as a consensus change to Litecoin. It certainly doesn’t matter that the price went up. (Accepting Alex Ao’s description of Bitmain “blocking” Segwit is effectively picking a side you like, but that is really, really not the point here. I would be making precisely the same argument if Segwit had not activated on Litecoin because Bitmain had had more hash power.)

And then you encourage Innosilicon to lobby the Zcash Foundation to allow ASICs. In my opinion, this is not something that as the CEO of ZcashCo --which is a role that already has a highly centralizing influence-- you should be doing. If Innosilicon want to lobby the Foundation they can do so; they don’t need your encouragement or endorsement.

[*] Granted, part of the reason they were able to do this is that Litecoin’s consensus change activation mechanism, inherited from Bitcoin, directly uses the proportion of mined blocks within a window to gate activation. Zcash doesn’t use that mechanism. But there are less direct ways in which hash power would still influence protocol evolution for Zcash and related coins.

31 Likes

This was an interview, it wasn’t active collusion. It was an attempt to increase transparency and we learned a lot: Innosilicon pre-mines the machines they are selling, they will mine secretly on GPU coins like Monero and that they can develop an ASIC in <=4 months. In short, Innosilicon seems like the evil twin of Bitmain. Who wants a used miner at a new miner price? Good wine ages well, not hardware.

he said that at one point the certain party that was blocking Segwit threatened to use its substantial Litecoin hash power, and he revealed that he had even more Litecoin hash power

The Segwit addition to Bitcoin was highly controversial that led to a hard fork which has yet to be resolved. Segwit changed the block fee dynamics, giving an artificial discount for Segwit transactions. It increased the Bitcoin blocksize in a very inefficient way while calling it a scaling initiative for off-chain (controversial) initiatives that were put forth by a for-profit company. Not to mention code bloat and technical debt. Preventing Segwit on Litecoin was common sense. It’s very concerning that Innosilicon is on the wrong side and getting involved with code changes. This is something Zcash Foundation and Company should be very cautious about.

5 Likes

I don’t see why the Foundation would be in a position to make a public statement about not changing the PoW or retaining Equihash (200,9), when they have only just started their work.

Also, how would Innosilicon “get them” to make a public commitment?

The business fortunes of Innosilicon, or any other hardware manufacturer, fall outside the purview of the Foundation’s mission.

10 Likes

Does nobody else have an issue with certain parts of this conversation.

Followed by this.

This is 2 asic producers deciding what Litecoin will do. Not the Litecoin community- 2 asic producers. He talks about making contributions and funding developers. This is a politically correct way of saying I bribed them to get what I want.

An asic company using its hashing power to control how a coin developes.

This is confusing. They only mine on the first batch for a week but have already been mining on them for a couple of weeks. Which statement is false.

9 Likes

The thing is, with Bitcoin and Litecoin, the most hashpower always determines what code is “Bitcoin” or “Litecoin” because the most hashpower produces the longest chain of work. This isn’t the case with Zcash due to trademarks.

Pre-2015, miners were more reluctant to get involved with taking a position on code changes for any cryptocurrency. Unfortunately that changed after the scaling debate, when a small few developers with “vetos” successfully blocked important upgrades with Bitcoin, minimizing it’s utility in the process. Miners started to get more involved at that point because they realized that developers were at risk of undermining the potential viability of cryptocurrencies just by making bad decisions.

So, I understand why it might seem scary that miners and developers of mining equipment would get involved with code, but the intent, generally, is to make sure the coins have a successful future, so that they can continue selling hardware for those coins.

3 Likes

I think that’s a second point here - as a manufacturer of mining equipment, you should be thrilled that a crypto changed its PoW algorithm as you get to make and sell another new product. As long as you made your profit on the first batch, why would you care? Only people actively mining should really give a damn if the PoW changes.

1 Like

I think this is a really good point. Depending on a lot of factors that I have gained only partial visibility into so far, changing the PoW could be:

  1. A gift to current GPU miners (both small/retail and large/industrial) who experience lower levels of difficulty (competition) at least temporarily,

  2. A punishment upon small or late-comer ASIC manufacturers who haven’t already made a profit or who have to invest too much of their working capital to spin up a new product,

  3. A gift to the largest and earliest ASIC manufacturers by harming or deterring their competitors without actually harming their bottom line,

  4. A punishment to small, retail ASIC miners who just bought a few units and haven’t earned back the costs yet,

  5. Harmless to large-scale and secret ASIC miners who already deployed many units and who have already earned back their costs.

If I’m right about even some of the above (especially point 3), then making changes to the PoW could potentially be the worst possible move in terms of driving out competition and rewarding the largest players, long-term.

Unless it’s not! It’s hard to tell. One huge factor that I don’t really have visibility into is what proportion of the current GPU mining hashpower is small/retail and what proportion is large/industrial. As far as I know the large industrial GPU miners operate in secrecy just like the large industrial ASIC miners do (or did until the recent disclosures by Bitmain and Innosilicon). I think it is quite plausible that the largest current GPU miner has 50% of the current hashrate. Or maybe not! Maybe it has only 25%. Maybe it has only 5%. How can we tell?

But the bottom line is that I believe it is entirely plausible that making ad hoc, quick changes to the PoW like Monero did might give a short-term boost to the current GPU miners while tilting the playing field toward worse centralization long-term.

I like that fact that Josh Cincinnati’s Simple ASIC-Friendly PoW deliberately takes those concerns into account.

5 Likes

Here’s an off-the-cuff idea: issue a public call to the large-scale industrial miners to publicly disclose their total hashrate, what blockchains and pools and transactions they’ll mine (e.g. will they mine empty blocks?), and whether they use GPUs, FPGAs, ASICs (and what model from what manufacturer) or something else.

Many such companies have privately told me that they have large-scale ZEC mining in the past. I could tell them that they should opt-in to disclosing that to the public as well.

Seems like they should be held to the same standards of transparency that we are currently trying to hold Bitmain and Innosilicon to.

4 Likes

DASH had promise until ASIC’s took over look at the trajectory now, don’t make that same mistake, in regards to Josh Cincinnati’s Simple ASIC-Friendly PoW, these are the airbrain schemes of lecturers/teachers who have no real world understanding and the complexities thereof that is why they teach.

This is a misnomer as installed base of GPU’s available to mine with right POW is many order of times more than any one manufacturer could have (GPU’ have been sold for many years), just look at Nicehash. Bottom line a POW can be developed which levels the playing field where ASIC’s should not have more than a 1.1x-1.2x advantage (look at ProgPow) over commodity hardware, which will be healthy long term, instead of locking into ASIC specific POW which then favors he who has the best manufacturing nodes and volume i.e. large manufactures, it would be no different than what is happening now in Bitcoin mining scene as the logical conclusion.

1 Like

Many of the pools show what addresses have the highest hashrates. Many of them don’t however. I remember watching the BTCP mining donation pool and there was a miner with around 1-2 Msol/s. That was the largest I had ever seen anywhere.That should have been pre-ASIC in February but maybe not. In most pools I have seen in the last 6 months the top miners had a few hundred thousand Msol. Those are moderately large, but I believe the multi-million Msol GPU mining outfits are rare or don’t exist on equihash. I think most equihash miners are small to medium sized.

2 Likes

You could easily argue that not forking is a gift to ASIC miners. Remember, that they choose to make ASICs for a coin that claimed to be ASIC resistant and claimed that it would likely fork if ASICs were made for Equihash. If anyone has a "right’ to mine Zcash, it’s GPU miners, not ASIC miners. (Ultimately, it doesn’t really matter who benefits – what matters is what is best for Zcash. My view is that GPU mining is better because it’s more decentralized and gets Zcash into more hands)

Second, it’s unfortunate that you spent a lot more time meeting with ASIC manufacturers than researching GPU mining. As I pointed out many times, most of the Zcash blocks until recently were found by small miners. An easy way to test this is to look at the blockfinders on a big pool like Flypool or Nanopool. They are almost always people with five rigs or fewer, which usually means it’s a home operation. Recently, however, nearly half of Zcash blocks have been found by one address that belongs to an ASIC miner.

12 Likes

Don’t think so much , just see the Bitcoin, completely centralized and controlled by a few hands in China, ¿ Is that what you want for Zcash ?..

damn it in his head , I love it Daira.

1 Like