Log in

entries friends calendar profile SMRT-TV (where my monthly column, Zoinks, appears) Previous Previous Next Next
Maybe it's just me - Si, and I like potatoes.
Maybe it's just me
But does this apology from Ustream for the Great Hugo Awards Debacle of 2012 basically imply that their engineers are completely incompetent? The money quote here is, "we were not able to lift the ban before the broadcast ended." That speaks to either bad technology or bad employees (noting that the former is often the result of the latter anyway, but might not be on the heads of the current employees).

It's generally a sincere apology, although there's a touch of victim-blaming in the statement about how folks who tell them they've got clearance in advance would have been fine (even though he admits the communication wasn't clear, and never even addresses the fact that the clips were probably legit by fair use standards anyway). But getting rid of Vobile is a damned good start.
11 comments or Leave a comment
jsburbidge From: jsburbidge Date: September 4th, 2012 12:59 pm (UTC) (Link)
I note that when I first followed a link to the apology, it had a reasonably large set of largely critical comments; it has now removed them and has comments turned off -- which feels to me like ducking feedback.
yendi From: yendi Date: September 4th, 2012 01:04 pm (UTC) (Link)
Hunh. By the time I saw it, the comments were already turned off, and I'd (foolishly) assumed that's how the post started. Gotta love a company that squelches criticism.
ckd From: ckd Date: September 4th, 2012 03:13 pm (UTC) (Link)
Maybe the comments were caught by Vobile and automatically removed. :-)
yendi From: yendi Date: September 4th, 2012 06:47 pm (UTC) (Link)
cheshyre From: cheshyre Date: September 9th, 2012 08:51 pm (UTC) (Link)
I already had another browser window open when they deleted the comments, so managed to screencap the text (though no formatting) of the 29 comments visible before the wipe:

29 Responses to Hugo Awards: An apology and explanation

Jennifer Smith says:

September 3, 2012 at 5:13 pm

"Unfortunately, we were not able to lift the ban before the broadcast ended. We had many unhappy viewers as a result, and for that I am truly sorry."

Really? Because the people running the event were explicitly informed that Ustream had no intentions of lifting the ban. So shouldn’t this read "we weren’t even trying to lift the ban before the broadcast ended."

I’m sure you are very sorry that people are angry with your service and now looking for alternatives.

Danny Ron says:

September 3, 2012 at 5:26 pm

Is this a jock? Why do you even use a 3rd party service? You don’t have any obligation to monitor the streaming of users! The copyright holder can monitor and file DMCA notice if needed. They didn’t? And even if you do use this bug-full 3rd party system, how come YOUR admin cannot override it? You need to "lift the ban"? It is your site. Just do it. This explanation makes little sense.

Marc Whipple says:

September 3, 2012 at 5:45 pm

Usually, this sort of thing might have worked, but you are not going to convince a bunch of angry science fiction fans to absorb some technobabble which is obviously an attempt to mislead them about what happened and your ultimate responsibility for the outcome. If you had done this to, say, the Lucies, it might have convinced somebody. Several thousand ticked-off science fiction geeks, not a chance. Do not pass Go, do not collect $200.

While pondering this event, assuming you are and some functionary tasked with mollifying social media didn’t ghostwrite it, consider that a VERY large portion of the technological community are science fiction fans. They are the ones who are needed to make things work. If they get the idea that Ustream isn’t reliable or responsive… draw your own conclusions.

Care to try again?

DMcCunney says:

September 3, 2012 at 5:59 pm

Thank you for the apology and explanation.

I was one of those who was watching the stream and saw the feed abruptly terminate.

The damage goes a bit farther than your stats might indicate. A friend was part of Chicon’s tech crew. Chicon was pulling back the feed to feed the overflow room where folks at Chicon who could not get into the ballroom were watching the ceremony. Chicon was also collaborating on cross-programming with Dragoncon, taking place in Atlanta at the same time, and Drqagoncon attracts 25K+ people. Many folks at Dragoncon for professional reasons had an interest in the Hugos, and were watching the stream. The UStream feed was what Dragoncon got, too. (The Chicon tech folks had no idea the feed had been cut till someone from the overflow room informed them.)

There are a lot of unhappy folks out there.

The explanation was about what the folks I talked to assumed was case – third-party monitoring software being overly aggressive. I understand why you need to use such tools, but any automated solution that can issue takedowns on its own has potential issues. As well as recalibrating the software, so it has a more accurate idea of fair use, you really need to review and adjust the procedures used when an error occurs. It’s good that your people tried to restore the feed when they were aware of the problem, but very bad that the ceremony was over before they succeeded. They should have been able to restore service almost as fast is it was cut off. Not being able to take prompt action when an error like that occurs in unacceptable in a service provider.

I have no idea what LoneStarCon, the 2013 Worldcon, plans to do for making the Hugo Ceremonies available, but I suspect "anyone but UStream!" will be the dominant attitude.

There are events that have prestige and influence well beyond their size, and the Worldcon is an example. I wish you the best of luck in preventing a recurrence of this problem, because I think you’ll need it.
cheshyre From: cheshyre Date: September 9th, 2012 08:51 pm (UTC) (Link)
Riiiiiight says:

September 3, 2012 at 6:02 pm

Save the doublespeak for someone who is dumb enough to believe you. It might not matter to you, since I am only one person, but I will NEVER use your service again, in any way, for any reason.

Anne says:

September 3, 2012 at 6:19 pm

Oh, UStream. You’re dead in the water, kids. This kind of thing gets around in the tech-y community. The people who would be recommending your service? Won’t be recommending your service.

romsfuulynn says:

September 3, 2012 at 6:32 pm

Your service blew it. Not only did you lose thousands of SF fans (who aren’t going to buy this) but these are people who are technophiles. I work for a federal government agency, and would be aware of this failure when considering any live event. I volunteer at a church denominational level that livestreams events. This is something that I’d bring to the discussion.

You need to do a lot better than this for an explanation.

Mark Richards says:

September 3, 2012 at 6:57 pm

I was wondering about the Dragoncon feed, and am dismayed to learn it also depended on UStream, along with the Chicon overflow itself.

Sorry guys, your explanations really don’t wash.

I accept that the brain-dead Vobile bots don’t know what fair use is.

What I don’t buy is that once Chicon people reached you and confronted you with the situation, you were unable to do anything at all to override what the stupid bots had done.

Ted says:

September 3, 2012 at 7:03 pm

Really? REALLY!

I have a feeling Ustream has just won the award for having the least amount of "tech cred". How do you mess up something like the Hugo’s? Because this "appologies" makes me think you have no scifi fans working at what is supposedly at TECH company.

1. It was made clear that the folks on staff had reached out to Ustream to deal with the situation. It was also made very clear that the feed would not be started.

2. Not only was the feed being used at ChiCon, DragonCon it was also being watch by those of us who actually had to work the weekend (yes, I do work for a tech company in Silicon Valley so while bring working on some new code I had this on in the background). We, the network folks in St. Louis and those of us in Mt. View where actually discussing the Hugo’s in a chat session. So see #1 how Ustream looses several geek levels.

3. To quote a responder above "do not pass go, do not collect $200?. Your "apology" may have worked on the cat channel folks (nothing wrong with watching cats on Ustream, I do it myself sometimes) but to use the excuse, or reason, of "third party software" doesn’t fly with the approx 5 to 6 thousand folks attending WorldCon (those figures I am basing off the numbers for ConJose, the WorldCon in 2002 in San Jose, CA) or the 25,000 folks at DragonCon. Some of those who are authors who have a vested interest in the results.

4. It is the responsibility of author/owner of copy written material to file a DCMA claim asking that the infringing material be removed. It is not your job nor the job of a "3rd party software"

Basically you failed Ustream, sack up and admit it.

R. King says:

September 3, 2012 at 7:49 pm

I don’t think even the most heartfelt apology or the most abject mea culpa is going to haul Ustream out of the fire. I’m not a sci-fi fan (well not enough of one to watch the Hugo Awards live,) and even I’ve been exposed to this fiasco. I know of a few organizations who’ve removed Ustream from their list of possible streaming services because of your demonstrated lack of foresight (imagine an awards show containing clips pf copyrighted shows —unthinkable!) and inability to control your own service.

In 6 months, you’ll be out of business, I think.

But what I really want to know is how you are going to make the Hugo Awards folks whole? An apology and a refund (you have refunded their money, right?) won’t cut it.
cheshyre From: cheshyre Date: September 9th, 2012 08:52 pm (UTC) (Link)
Matthew B. Tepper says:

September 3, 2012 at 7:49 pm

I do not accept your apology.

Let me spell it out for you in words that even a professional PR hack could understand. You ruined a live event. Not just A live event, the live event of a particularly unforgiving bunch of people. You want to know just HOW unforgiving? Well, let’s see….

The Starlight Motel in San Diego was the venue for the Westercon (Western Regional Science Fiction Convention) in 1966. That’s right, forty-six years ago. Fans still sing a song ("Bouncing Potatoes," words by Poul Anderson, to the tune of "Waltzing Matilda") about the awful banquet food at that convention —with an oblique mention of the call girls who were tolerated by the motel in return of a cut from their "business."

Okay, that’s one example, but I can think of another, occasioned by a moment which was carried by your feed before your stupid, badly-programmed bots pulled the plug on it. The winner in the category of Best Editor, Long Form, was Betsy Wollheim. This was the first nomination, and first win for Betsy. It was not the first nomination within her family, however; her father, Donald Wollheim, was a professional editor and fan going back to the 1930s. In 1939, Wollheim père and other members of his fan group, the Futurians, were ejected from the (first) World Science Fiction Convention in New York City. Some remnants of that grudge may have been carried on through to the next generation, as Betsy mentioned that her father had finally been vindicated with her win of this award.

From the 1939 Worldcon to the 2012 Worldcon —that’s seventy-three years, to save you the trouble.

So, science fiction fans have very long memories, and grudges can last a very long time. This is not going to just be forgotten; Ustream is going to be associated with "FAIL" in the fannish memory for a very long time. Perhaps the rest of my life. Perhaps even the rest of yours.

And don’t think that science fiction fandom is just going to let this slide without telling anyone. Some of us can be strident, vindictive, and very persistent.

I don’t say that it wouldn’t help if your apology took some sort of financial form as well — say, some very large donations to future Worldcons (and certainly something for the one just ended) sponsorship of science fiction exhibits at university libraries, and a hefty gift to the EFF, so that they can fight inanities such as excessive enforcement of the vile DMCA.

I am speaking only for myself, of course, and not on behalf of any organization of science fiction fans. I merely speak as a science fiction fan with some knowledge of our history, as well as an obvious sense of outrage with incompetence and half-hearted apologies.

You have some hard work ahead of you. Get to it.

Dale says:

September 3, 2012 at 7:53 pm

Well, it’s lovely that you are sorry about your colossal screwup. You pretty much just pissed on the parade ground in front of the commanding general. You used an overly aggressive content screening service, and you allowed this overly aggressive system that you don’t have control over to automatically take down feeds. You failed to delegate authority down to the people who were on duty in your control center to fix the situation in a timely fashion.

I know that I will not be recommending ustream for any of my customers for anything that actually needs reliability. If I hear of a customer or friend who is planning on using ustream for anything that is in the least bit important to be delivered in a timely fashion, I’ll be telling them of this incident as a cautionary tale.

Matthew B. Tepper says:

September 3, 2012 at 7:54 pm

Yep, I slipped an HTML tag. How embarrassing! That, however, will be forgotten in a week. On the other hand, I just made a typo, while your lack of a preview mode suggests lack of technical expertise on your part.

Robert J. Sawyer says:

September 3, 2012 at 7:54 pm

You write, "As background, our system works like this in order to support a large volume of broadcasters using our free platform. Users of our paid, ad-free Pro Broadcasting service are automatically white listed to avoid situations like this and receive hands-on client support."

Perhaps not quite the right time to be trying an upsell, guys …
cheshyre From: cheshyre Date: September 9th, 2012 08:52 pm (UTC) (Link)
Baylink says:

September 3, 2012 at 8:31 pm

Having worked in broadcasting on several levels over a 3 decade career, I have several observations to make here:

1) If you couldn’t unban it, then someone (probably several layers of someone) is incompetent to do their job and should probably be fired; the higher up they are, the more likely this is to be true.

2) If you don’t have a realtime status display in your NOC of channels by descending order of active viewers, see point 1.

3) if you *do* have such a display, it’s probable that any automated processes you have for things like content "protection" should go into manual mode when a stream hits the top 40; the NOC should be warned, and someone in authority should manually make a call.

Y’know, like in the Heidi Game.

and finally 4) you should have a real time communications system for the broadcasters in that top 40 to get to your NOC and your administrative staff so as to get things like this fixed.

You were the ones who asked to become a TV network. You got it.

Now act like it.

(Oh, and having a comments page that works reliably would be good too… Second try)

Wzrd1 says:

September 3, 2012 at 9:10 pm

In other words, "We don’t know how to work the thing, so we’ll have to wait for Vobile to open on Tuesday". Geeze, I think I’ve been contracting too long. I’m becoming too fluent in PR speak.

Kenneth Thomas says:

September 3, 2012 at 9:17 pm

Thanks Brad. I know others will not agree with me, but I appreciate the apology, explanation and clarification. It seems significant that the Hugos could have paid for premium service, and that you are re-calibrating the free service. I hope others will take this into account.

Wzrd1 says:

September 3, 2012 at 9:48 pm

Kenneth, that seems excessively company pro and not considering the other clients who utilize the service without paying a fee. You’re saying, we’ll fix it if you pay us and screw you if you don’t. That is one of the lousiest business models I’ve ever heard of for a service like this. As in business crushing, client alienating crushing.

But, since you got my attention during my random page refresh interval, one other potential point of failure is likely. Management not permitted to make a decision to pull the unnecessary "protection" (DCMA doesn’t harm providers, such as Ustream unless action isn’t taken AFTER notification of a violation). As in micromanaging the company, not permitting managers to make a decision. It’s that or incompetence in the NOC or failure to follow due diligence in the operation of the third party provided system. If it’s the former, it can be easily fixed by a mental change at the top. If it’s the latter, well, I’ve fired better many times in the past. During this depression, replacing the incompetent is trivially easy.

Adam says:

September 3, 2012 at 10:06 pm

Wow, is there a lot of self-importance in this thread. The Hugo stream is unlikely to have been anywhere *near* the top 40. It pulled in very little money, and yet is causing an inordinate amount of trouble. One person has even suggesting extorting money out of the company by threatening to persistently badmouth them. Overall, the commenters here sound like a bunch of petulant children. You guys represent some of the worst of fandom.

Leon Hunter says:

September 3, 2012 at 10:50 pm

What an absolute Joke, hilarious on every level.

I’ll tell you what, lets say I run a review stream that reviews movies and I display 30 seconds of these movies to the context of reviewing it or even a two minute trailer.

You’re telling me that in your system, you find it perfectly acceptable that some automated process would kill off my stream for copyright infringement, when it is well within the doctrine of fair use?

Ustream will have an infamous reputation for this outrage.

Adam says:

September 3, 2012 at 11:08 pm

@Leon: A full-blown trailer would almost certainly be considered to be a "substantial" part of the work (since it will usually aggregate many of the main plot elements), and therefore would not be considered fair use. So, actually, their system would be "right" in stopping your stream.
cheshyre From: cheshyre Date: September 9th, 2012 08:53 pm (UTC) (Link)
johnson says:

September 3, 2012 at 11:33 pm

So basically the Hugo Awards opted for the free service instead of the paid service. A bunch of people watching for free were annoyed when the free broadcast was terminated due to controls that would not have been in place, had the paid-for service been used. I guess they could always ask for their money back? A FULL REFUND!!!

Yoz says:

September 3, 2012 at 11:52 pm

I’m with Adam and johnson on this one. If the Hugo Awards stream is important enough to get so many people so angry, how come it’s not important enough for the organisers to pay for a service level agreement?

It’s the Hugos organisers that people should be screaming at, not UStream. They picked the wrong streaming arrangement; instead of paying up front, they paid with interrupted service, and the fans paid too.

Mark Vaughn says:

September 4, 2012 at 12:01 am

When I was told about this catastrophe, I had no words. UStream has been in the live streaming business for years. Not only that, they have millions in funding to run this site. More than enough resources to make sure everything is in proper working order.

You see, I am the founder of a competing live video site. I will not say which site, but you can probably figure it out based on my last name, Vaughn.

Every broadcast, especially award ceremony broadcast, should automatically be given special care. I’m not saying they’re above any other broadcaster, but when you’re dealing with high traffic, or even potential high traffic scenarios, it’s best to keep an extra eye on things to make sure they go smoothly.

Don’t tell them to buy a package to avoid issues, that is a major cop out and speaks volumes to the laziness and/or unwillingness of creating a great experience for your viewers and broadcasters.

As a competitor to UStream, I feel UStream has failed incredibly to protect its broadcasters from situations such as this. I’m not saying my service is better, I’m saying you be the judge of what is the better experience for you.

Clearly, UStream is not too concerned.

- Mark Vaughn

tiggy says:

September 4, 2012 at 12:09 am

Voz, Adam and johnson. I assume the free stream is intended to provide free publicity for the product. (Probably cheaper than paid advertising?)

And it did.

Pillow top mattress says:

September 4, 2012 at 12:13 am

I hope others will take this into account.

Wolf Baginski says:

September 4, 2012 at 12:26 am

So, let me get this right. You use a third party to detect possible copyright infringements. The Worldcon streams a live event , which makes use of material provided by the assorted copyright holders, pretty much as happens at any awards ceremony. That third party manages to spot these clips, and kills the stream.

I must be badly out of date on that, because, the last I heard, matching video clips to programs was computationally difficult. There are some huge amounts of data involved. It occurs to me that the system may have been detecting something no more significant than a broadcasting company’s logo. If so, you are buying in a service which might not be doing what you think it is.

As for your reputation, personal and corporate, you’ve just messed up a key annual event for a culture which is generally technically competent, and which can remember the strangest things. You’re falling headlong into myth and legend. You face becoming the "big bad" of a story told to children.

Adam says:

September 4, 2012 at 12:28 am

@Mark: I might have agreed with some of what you wrote, except for that little tiny bit about ChiCon/WorldCon itself relying on Ustream to display the presentation in their own overflow room. If your event is relying on the service for your own attendees, there is no excuse not shell out a few bucks to use the non-free version. Or at least ask for an in-kind donation of the service.

And you’re still perpetuating the fallacy that this was a "high traffic" scenario. The number I saw elsewhere was that it was around 650 connected clients. That’s peanuts.
cheshyre From: cheshyre Date: September 9th, 2012 08:53 pm (UTC) (Link)
The Good Knight Irene says:

September 4, 2012 at 2:25 am

@Adam: It’s true the number of "clients" watching Ustream online was around that. I was one of them, and there were four people watching our feed. We had been watching the number of viewers and overall views tick up and up. There were over 700 viewers and the number was going up rapidly as Neil Gaiman’s acceptance speech began. The number of views, which is not the same thing, was somewhere around 1,000-2,000.

But this is a community which cares deeply about these things, is tech savvy, and talks to each other.

Furthermore, I do not know how many of the 6,000 Chicon members or 25,000 Dragoncon members were watching the ceremony through feeds. Those would have shown up online as only one "viewer."

Many of the people who were not watching the broadcast were the important movers and shakers who were actually in the ceremony themselves. I doubt they are happy about this.

That a relatively small number of people were viewing the fiasco live does not mean this will have little impact.
PS: I agree that this comments interface is terrible. My view window is cut off on the right so that the last few letters of each line are cut off. The words in the captcha window are also incompletely viewable so that I shall have to try again. And I have tried to post this four times so far.
yendi From: yendi Date: September 11th, 2012 12:13 pm (UTC) (Link)
Awesome! Thanks for capturing this!
11 comments or Leave a comment