Common Downtime O1 FTSO, Fly2Sonic, Flare Ocean, Civil FTSO, Anon2

[Neil_AU]

Provider Names: O1 FTSO, Fly2Sonic, Flare Ocean, Civil FTSO, Anon2
Provider Delegation Addresses:
O1 FTSO: 0x229458a754cd1aeba8a0c87f59e22777d593b85a
Fly2Sonic: 0xe6caa2bca8b0e9004724e0600b38db52c17bcac30
Flare Ocean: 0xf4213e49488b9320769d35924ac52ea31a4c9fc1
Civil FTSO: 0x733a73408d8e106b6ac999ccd9f3d308bcaeb7b5 (already voted on, awaiting Foundation decision)
Anon2: 0x814cd7d3cee516b8e0537085ecdeca7695d65c63 (already voted on, awaiting Foundation decision)
Network: Songbird

Overview: Common downtime among multiple (5) providers suggests they’re using the same infrastructure / data source / api.

Evidence: Two pieces of evidence: the first shows downtime observed among the named providers, all starting within one epoch of each other and lasting many hours before returning independently. All 5 providers submit epoch 458607, 3 providers submit epoch 458608, all providers then experience downtime for ~5hrs before coming online independently over the subsequent 10hrs.

https://songbird-ftso-monitor.flare.network/price?currency=BTC&startTime=1d&providerAddress=0x229458a754cd1aeba8a0c87f59e22777d593b85a,0x733a73408d8e106b6ac999ccd9f3d308bcaeb7b5,0x814cd7d3cee516b8e0537085ecdeca7695d65c63,0xe6caa2bca8b0e9004724e0600b38db52c17bcac3,0xf4213e49488b9320769d35924ac52ea31a4c9fc1&endTime=1714457611&relative=false

Downtime observed on all symbols, BTC in the above example and XRP in the below.

https://songbird-ftso-monitor.flare.network/price?currency=XRP&startTime=1d&providerAddress=0x229458a754cd1aeba8a0c87f59e22777d593b85a,0x733a73408d8e106b6ac999ccd9f3d308bcaeb7b5,0x814cd7d3cee516b8e0537085ecdeca7695d65c63,0xe6caa2bca8b0e9004724e0600b38db52c17bcac3,0xf4213e49488b9320769d35924ac52ea31a4c9fc1&endTime=1714457611&relative=false

The second shows Fly2Sonic and O1 FTSO alternating their submissions over a period of approximately 10 hours, with neither submitting at the same time.

https://songbird-ftso-monitor.flare.network/price?currency=BTC&startTime=1d&providerAddress=0x229458a754cd1aeba8a0c87f59e22777d593b85a,0xe6caa2bca8b0e9004724e0600b38db52c17bcac3&endTime=1696275581&relative=false

Both providers last submit together at epoch 357693.

Fly2Sonic continues submitting until epoch 357770, O1 FTSO does not submit during this period.

O1 FTSO then starts again at epoch 357771 and Fly2Sonic stops submitting.

O1 FTSO continues to submit until 357879, neither provider submits epoch 357880.

Fly2Sonic submit 357881, O1 FTSO do not. Neither submit 357882.

O1 FTSO submit from epoch 357883, Fly2Sonic do not.

Both resume submitting from epoch 357908.

In both pieces of evidence the suggestion is these providers are sharing infrastructure, data feed or using the same pricing api.

1st Offence: Flare Ocean, Fly2Sonic.
Previously Chilled: O1 FTSO once. Civil FTSO and Anon2 chilled once with a ban decision pending.
Previous Chill Proposal Number: O1 FTSO (SMGP-14), Civil FTSO (SMGP-16, SMGP-24), Anon2 (SMGP-17, SMGP-25).

Timeframe for Open Discussion: 7 days

Request of Management Group Members: I respectively asks all MG members for input on the evidence provided in this proposal and to participate in subsequent on-chain voting following the discussion period. Depending on the feedback from the providers mentioned, proposals will be created for O1 FTSO, Fly2Sonic and Flare Ocean.

Request to the Flare Foundation: If voting meets quorum, I respectfully request that the Foundation action a two epoch chill of Fly2Sonic (0xe6caa2bca8b0e9004724e0600b38db52c17bcac30) on Songbird, Flare Ocean on both networks (0xf4213e49488b9320769d35924ac52ea31a4c9fc1 & 0x50bd2D0B457E27a481f97C4685d16EeB8B978155) and a permanent ban of O1 FTSO (0x229458a754cd1aeba8a0c87f59e22777d593b85a & 0xBE304C28F3a050486b9733AE56cB5541B16c007B) on both networks.

These actions should apply to both FTSO V1 and V2.

Both O1 FTSO and Flare Ocean have registered for V2, at this stage Fly2Sonic have not.

O1 FTSO

https://www.useyourspark.com/analytics/monitor

Flare Ocean

[Brent-4Dads]

Well done. Can’t imagine the time it took to find these correlating down times. Though we have suspected these guys share infra, and don’t care to support the network in that way, it’s nice to see evidence supporting it. Interested to see their response.

[Neil_AU]

Please review and comment @O1FTSO, @Fly2sonic_ftso, @FlareOcean.

[AureusOx]

Well done @Neil_AU. This is some clear evidence, especially so given that a few of these providers have been previously chilled.

[AtlasTSO]

GJ Neil and thank you for your work.
To expand on this did not find any significant outages in the same timeframe for the rest of the providers or any notices from the known and big Cloud or dedicated providers for outages.
This kind of evidence speak to resource sharing a lot more than price spikes.
Looking forward to hearing from those included.

[O1FTSO]

Thank you for your hard work, Neil.
I want to make it clear that O1 provider has never shared any algorithms or nodes. I am sure you will never find any collusions on the songbird monitor. I would like to inform you that this occurred because most providers mentioned above are using a hosting service called PowerVPS. We also use PowerVPS, and when the incident occurred, our songbird node was down for several tens of minutes. I am willing to share the source code and songbird node with a member of the Flare team to solve the problem. This absolutely cannot be evidence of collusion.

[Fly2sonic_ftso]

Yes, I admit it and realized the fact after the incident occurred. At the time, I was using PowerVPS hosting service for my infra, and there was a critical service downtime. I couldn’t connect to any of my servers as even ping requests were not working. The downtime lasted for about 3 hours after I reported it, and I noticed that other providers like O1 and FlareOcean were also experiencing similar downtimes. I suspected they were using the same or similar hosting platforms for their nodes.

Another issue I faced during that time was that I had not updated the event fetch code to the latest version—I was binding old events, which, strangely, had worked until then. After fixing the code, the provider came back online.

ftsoContracts.forEach((contractWithSymbol) => {
contractWithSymbol.contract.on(
“PriceFinalized”,
async (
epochId,
price,
rewardedFtso,
lowIQRRewardPrice,
highIQRRewardPrice,
lowElasticBandRewardPrice,
highElasticBandRewardPrice,
finalizationType,
timestamp
) => {

I am not sure what happened at the time of your second evidence since it is from long time ago.
In those days, I was updating the codebase hard since the SR was not reaching to my expectation.
I can say it is just a coincidence.

Thank you, @Neil_AU . I anticipated that this question might come up one day, and I’m happy to provide a reply.
Feel free to ask all of your doubts, and I am open to talking with everyone.

[AtlasTSO]

Thanks for answering.
If that is case then nothing to worry about.
If i were you i would ask powerVPS to confirm u are their client on that specific date and confirm you had issues and forward it here.

[FlareOcean]

Since community knows the whitelisted songbird node IP of each provider, its untrustworthy to say that a node has been shared. It is also possible to track IP addresses to find out where and for what services they are actually using servers. Our provider and nodes are also hosted on PowerVPS. The thought that this is evidence of collusion is highly inaccurate and biased.

[FlareOcean]

I want to share the screenshots I took at that time and the error was already shared with Tom on the 29th of April.

[Brent-4Dads]

How was your issue resolved. The others appear to be claiming it was the fault of power vps having an outage. Is that your claim as well?

[FlareOcean]

When the problem occurred, I had to check the status of the node and then rerun the script to run a node. Eventually, the node started working normally, but I think this may be a problem with the hosting service.

[Brent-4Dads]

Can you reach out to your hosting service to confirm they had an issue at that time and post screen shots h

[FlareOcean]

I have already contacted the support team.

[Brent-4Dads]

Let us know how they reply.

[FlareOcean]

Lol, I shared the screenshot with them and they just replied.

"You are asking us why an external resource to which we have nothing to do was not connected.

Contact the administrator of the server to which your server was supposed to connect and find out why it was not responding to the request.

Do you understand the essence of the problem?
Similarly.
Facebook does not work from your server and you are asking us why the server does not connect to Facebook. Do you understand?"

[FlareOcean]

Anyway, it is absolutely clear that there was an issue with the network.

[Brent-4Dads]

So now you’re saying it was the songbird network that was the problem? And it just impacted the providers in question? @Fly2sonic_ftso @O1FTSO do you agree with this shift of blame?

[Brent-4Dads]

Any way you can share your correspondence with power vps so we have a clear picture of all this?

[Neil_AU]

How do you explain the alternating of submissions for ~10hrs between your provider and that of Fly2Sonic in the second piece of evidence?