Jekyll2022-11-28T15:21:12+00:00https://labs.ft.com/feed.xmlFT Labs BlogExploring new technologies and ideas to help get FT Journalism to where it is needed.
[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]Traversing the data mines2020-02-17T08:30:30+00:002020-02-17T08:30:30+00:00https://labs.ft.com/research/2020/02/17/myft-machine-learning<figure
>
<img alt=""
src="/assets/uploads/2020/02/myft_ml_header_graph.png"
width="80%"
class=""
/>
</figure>
<p>Interested in the field of Machine Learning and guided by the FT's goal of adding further value through personalisation, we saw an opportunity to have a go at creating our own FT Labs content recommendation engine™. Our early estimates warned this could be quite difficult, and we were right.</p>
<h2>What did we do?</h2>
<p>Data is the cornerstone of any Machine Learning project. If you don't spend the time to gather, prune and enrich your data you will be left with low quality data - <a href="https://en.wikipedia.org/wiki/Garbage_in,_garbage_out" target="_blank">Garbage in, garbage out</a>. Through all of our reading and watching on how to get started with ML projects it was greatly impressed upon us that understanding the data and its context is essential.</p>
<p>The best metrics for evaluating/calculating personalisation would be any data points generated by a user. Therefore we would need to get hold of select user behaviour data sets, not currently open to us. Namely:</p>
<ul>
<li>Follows/unfollows - user action of choosing to follow an FT topic or the reverse</li>
<li>Saves/unsaves - user action of choosing to save an article for later or the opposite</li>
<li>Reads - articles a user has read</li>
</ul>
<p>Fortunately we were granted the keys to the FT shared large data solution™, unfortunately there are so many datasets contained within it was hard to know where to start.</p>
<figure
>
<img alt=""
src="/assets/uploads/2020/02/myft_ml_hanger51.jpg"
width="80%"
class=""
/>
<figcaption>Hangar 51 from Inidiana Jones - large repository storage of mysteries and wonders</figcaption>
</figure>
<p>FT Data teams to the rescue! Members of the Data Science and Data Support departments became our trusted guides, leading us to the data we sought. Data Science also introduced us to R which was invaluable in processing and manipulating data of this size. We also built upon Data Science's wealth of tools to get our own scripts up and running.</p>
<p>These data sets were immense. Over a number of weeks we looked at follows/unfollows and saves/unsaves in turn.</p>
<h2>User topic follows</h2>
<p>We discovered with user topic follows that we could use <a href="https://en.wikipedia.org/wiki/K-means_clustering" target="_blank">k-means clustering</a> to group topics together, based on the users who follow them, and allow the ML to make groupings for humans to label later. We initially wanted to group several types (people, topics etc) together but the compute costs were too high for our initial investigation, so we settled for topics (for now).</p>
<figure
>
<img alt=""
src="/assets/uploads/2020/02/myft_ml_clustergroups.png"
width="80%"
class=""
/>
<figcaption>FT article topics grouped together in clusters by a k-means clustering algorithm examining vectors of users following each topic.</figcaption>
</figure>
<p>Depicted in the image above are the 3, 5, 10 & 20 cluster grouping versions of FT topics. Cluster numbers were selected by using the <a href="https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set" target="_blank">elbow method</a> of estimating the optimal number of clusters in a collection of data.</p>
<p>Each version is the number of groups a human gave to the k-means algorithm and said "put all the topics into this number of groups". Each users follows are used as vectors to calculate the topics position, allowing similar patterns of topic following to be detected by the algorithm.</p>
<p>You can see that some groups change quite a bit as the number of clusters go up however others (like the trapezoid in the bottom left) are so relevant and separate from other topics that they hold onto their group cohesiveness even as the number of clusters increases.</p>
<p>This gave us a collection of topic groups that we could use to suggest new topics to users. "You follow this topic but what about the others in the group". While very useful, these clusters only represent one part of a recommendation, we'd need to look deeper to refine the recommendations.</p>
<p>As we further understood the data and improved our R skills we took a look at follows vs unfollows. This would turn out to take a lot more data massaging to ready the data for analysis.</p>
<h2>Follows/unfollows</h2>
<figure
>
<img alt=""
src="/assets/uploads/2020/02/myft_ml_follows_unfollows.png"
width="80%"
class=""
/>
<figcaption>FT concepts positioned according to number of users follows and user unfollows within a selected time frame, points coloured by the type of concept</figcaption>
</figure>
<p>We mapped the number of user concept follows to the number of user concept unfollows to see if we could identify concepts that had a stronger (higher) follow to unfollow ratio count. Armed with this ratio count we could rank the clustered topic suggestions in order of other users voting with their digital feet.</p>
<p>The above graph helped us determine if a topic was more 'sticky', meaning it retains its gained users more than other concepts. Both axes are scaled by log10 to help visually spread out the interesting portion of the data. Somewhat annoyingly, the most dramatic portion of the graph, looking like a fragment of X ray crystallography, is simply an artefact of the log-scaling, and only covers topics with the fewest followers (and therefore of least interest).</p>
<figure
>
<img alt=""
src="/assets/uploads/2020/02/myft_ml_follows_unfollows_guides.png"
width="80%"
class=""
/>
</figure>
<p>The light grey line is x=y, so any concepts on this line have exactly the same number of follows as unfollows. The black line is approximately marking the average 10:1 ratio of follows/unfollows each concept has. The further to the right of the black line a concept is, the greater proportion of follows it has.</p>
<p>This is far from a perfect ranking system. We discovered that due to some differences in tracking organic follows and organic unfollows we were not able to count organic unfollows with 100% confidence. Leading to concepts appearing on the left of the follow=unfollow grey line, which should not be possible.</p>
<p>We also discovered that while some topics have greater follow ratio, there could be other reasons not to push that to the top of the recommendations pile. Like when was the last time an article was published for this concept? Also there may be acceptable reasons for a large number of unfollows, for example the topic 'World Cup 2018' has seen a reduction in follows because the event has concluded.</p>
<p>This follows/unfollows ratio data gave us the first ranking filter we could use in the proposed recommendation algorithm. Now it was time to tackle our first crack at the FT Labs recommendation engine, known as Suggestr.</p>
<h2>Suggestr</h2>
<p>We built Suggestr as an endpoint that takes one or more FT topics as input and can suggest other topics of interest. We also included the correlation service, which also powered the <a href="https://labs.ft.com/games/2017/09/18/make-connections.html" target="_blank">correlations game</a>, so that we could find topics that appear in the same cluster as the input topic(s), exclude topics that directly correlate and then rank the remaining with their follows/unfollow ratio. This way we suggest topics that don't appear in the same articles as the input topic but have been determined (by ML analysis of user interactions) to be of similar interest.</p>
<p>So <a href="https://github.com/ftlabs/suggestr" target="_blank">Suggestr</a> was born.</p>
<figure
>
<img alt=""
src="/assets/uploads/2020/02/myft_ml_suggestr_output.png"
width="80%"
class=""
/>
<figcaption>Example screenshot of Suggestr's output when providing Blockchain as an input topic. Core results are in the results object, the rest of the output shows the steps taken to get the results, as not to create too much of a black box recommendation.</figcaption>
</figure>
<h2>Suggestr's mission</h2>
<p>Using topic cluster analysis, topic correlations data and topic follow/unfollow ratios - can we suggest retrieve topics to follow (or retrieve articles) for based on any given topic(s)?</p>
<p>The first prototype of Suggestr works well but we'd like to refine and add to the ranking filters before user (reader) testing can begin.</p>
<h2>What did we learn?</h2>
<p>All those ML & AI experts saying data prep is the largest time consumption percentage of the project weren't kidding. However all that time spent familiarised us with the data and allowed us to build effective tools to minimise time spent in future revisits.</p>
<p>If we don't record the data we have in a documented or understandable way that priceless data is basically number garbage. You need to be able to trust your data, to trust your analysis for your users to trust your output. So let your use of the recorded data shape the future recorded data, a cycle that will continue to improve your data (and projects) into the future.</p>
<blockquote>
<i>"Every company has messy data, and even the best AI companies are not fully satisfied with their data."</i> - <b>Andrew Ng</b>
</blockquote>
<p>It is difficult to generate recommendations based on a few metrics, you'll want as many as you can to shape the best suggestion. In the end there may be no good suggestions right now, and it's okay to say that rather than making a bad/false suggestion and losing the trust of a user.</p>
<p>Our bumbling around in the data helped other teams understand how tricky navigating it can be without help. Efforts are already underway to clarify and clean the data for everyone's use.</p>
<h2>What does the future hold?</h2>
<p>While currently on hold we look forward to completing Suggestr in the future and testing the quality of its recommendations against a live audience. There's an interest here at the FT to find recommendation models that fit each user best, or that users can pick and choose from.</p>
<p>We've also come a long way into understanding the capabilities and limitations of not only ML but our own data. We now have access to large relevant datasets with multitudes of possible investigations to help discover and support future projects.</p>
<p>We'd call that a win.</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]Investigating FT user dataAntagonising Lucy Kellaway with Experimental Amy2019-08-28T14:55:30+00:002019-08-28T14:55:30+00:00https://labs.ft.com/experiment/2019/08/28/antagonising-lucy-kellaway<p>At 3.51pm on 2017/05/17, a Slack message popped up: …</p>
<blockquote>
<p>“What’s the best way for us to prevent this happening? <article link><br />
We have the article plastered with your audio thing<br />
but she always does her own podcast version.<br />
<br />
… <em>and she does not like this one bit</em>.</p>
</blockquote>
<p>With that, it looked like our foray into artificial voices, might be over. Weeks of work. First day live.</p>
<p>Lucy Kellaway, <a href="https://ig.ft.com/sites/guffipedia/">scourge of corporate bullshit</a>, <a href="https://twitter.com/lucykellaway/status/696392193884475392?lang=en">doyen of the well placed barb</a>, <a href="https://www.ft.com/comment/lucy-kellaway">opinionated Columnist</a>, and writer that day of new column piece, had just noticed our audio experiment muscling in on brand Kellaway.</p>
<p>By chance, the FT Labs Team’s new audio prototype had selected the latest Lucy Kellaway article as one of the first assortment to be converted to audio using an artificial voice, Experimental Amy. The new audio player component sat prominently near the top of each article, giving the unforewarned <a href="http://www.ft.com">ft.com</a> readers a chance to listen to it.</p>
<p>Only Lucy Kellaway narrates Lucy Kellaway’s articles (<a href="https://www.ft.com/listen-to-lucy">Listen to Lucy</a>), so almost immediately we had fallen foul of the second of our two primary project don’t-do-thats:</p>
<ol>
<li>Do not besmirch the FT brand</li>
<li>Do not antagonise journalists (for they are quick to anger and then express that anger in beautiful, cutting prose)</li>
</ol>
<p>(Only then could we safely continue to explore the question: Is this good enough and useful enough for our subscribers?)</p>
<p>As we mobilised to limit the damage, and drop that one article from the experiment (which <em>should</em> have been an easy thing to do, you might have thought), an email arrived. It provided more context for the explosion behind the scenes (in part, years of frustration with the labour-intensive, error-prone podcast process), but with a new directive: Don’t take it down - leave the Amy audio in place. Lucy was now ‘intrigued’.</p>
<blockquote>
<p>“I would like to make it the subject of next week’s column. Do you have time for a chat?”</p>
</blockquote>
<p>The very next day, fully prepped, with assorted sales pitches and justifications at the ready, we went downstairs to do chat.</p>
<p>Lucy Kellaway was… lovely - interested in, and even supportive of, the Labs experiment. But also a journalist. The interrogation began.</p>
<p>(Some dramatic license may have been taken with the following transcription.)</p>
<h2 id="whats-experimental-amy-all-about">“What’s Experimental Amy all about?”</h2>
<p>Originally, we had a request from our Business Development team to help them gather enough supporting evidence to establish a business case for paying for human narration of FT articles. The Labs team built a framework to import narrations from a partner 3rd party audio provider, with a stable of mellifluous voice actors, and expose them via an audio player on the relevant articles on <a href="http://www.ft.com">ft.com</a>. The delays between article publishing and the arrival of a new audio file were of the order of 12 hrs. The audio was of high quality, and whereas the experiment involved just 3 articles per day, it would obviously incur a non-trivial cost if rolled out across the entirety of each day’s publications.</p>
<p>A year or two before that, there had been a brief foray into the use of automated voices in cars, but the voice quality at the time was awful, so that idea was parked.</p>
<p>With the human voiced articles experiment underway, we had another quick look at the state of artificial voices and were pleasantly surprised at how much better they now were. It was relatively easy to build a new ‘provider of audio articles’ using artificial voices, a pretend 3rd party, and plug it into the existing framework.</p>
<p>The audio experiment had pivoted.</p>
<h2 id="how-did-you-choose-the-amy-voice">“How did you choose the Amy voice?”</h2>
<p>We compared the voice offerings of the main providers at the time, Google, IBM, Amazon. Largely due to convenience and a wide choice of nice, interesting voices, we settled on Amazon Polly, which had recently been <a href="https://www.recode.net/2017/4/12/15274220/jeff-bezos-amazon-shareholders-letter-day-2-disagree-and-commit">plugged by Jeff Bezos</a>.</p>
<p>Of the Polly voices, Geraint was the clear team favourite. Experimental Geraint had the soul of a poet. He elevated mere reading of text into a lyrical performance, especially once we got him to read the lyrics of Bohemian Rhapsody. That audio file was all it took to convince any doubter that the quality of automated voices had now achieved a threshold of “wow, rather good”.</p>
<p>Since this was a public-facing experiment going to actual, subscription-paying consumers, we needed to be sure we had buy-in from the rest of the company, especially Editorial, Brand, Marketing, Product. Until that point, no-one had asked (or answered) the question, “What does the FT sound like?”</p>
<p>We conducted numerous auditions, narrating FT articles with a variety of automated accents and variants: English, Welsh, US, Indian English, Australian, male, female, and French and German (because, why not?), all offered by the Polly API. The unanimous choice was Amy, a clear, boring, english, female voice.</p>
<p>[This in turn highlighted an issue with the human-narrated articles, where the voice actors were emoting that bit too much. More like reading a story than a news item. Amy did not emote. News was news, not Jackanory.]</p>
<h2 id="why-my-article">“Why my article?”</h2>
<p>Our initial approach was to use, somewhat arbitrarily, the articles cited in the daily <a href="https://www.ft.com/firstft">firstFT newsletter</a>, for narrating by Experimental Amy. On the first day of the experiment, firstFT just so happened to include one of your articles.</p>
<p>The first few weeks of the experiment are intended to consist of a few new articles each day with audio available to all readers.</p>
<p>[Later this changed to all articles for some readers, as we ran an A/B test.]</p>
<h2 id="how-long-does-it-take-to-create-audio-for-an-article">“How long does it take to create audio for an article?”</h2>
<p>We used the Amazon Polly API, which was very quick. That API had a rather frustrating limit of 1500 characters per request, which was fewer than in most FT articles, even the short ones. As part of the hackery to get the experiment running, we had to break up articles into shorter segments, finding the last convenient paragraph break just before the 1500 char limit, generating audio for each segment via the API, then stitching the audio segments back together into a single file. That whole process takes about 1.5secs per Lucy Kellaway article, and 3-4 seconds for a <a href="https://www.ft.com/the-big-read">Big Read</a>.</p>
<p>This is easily fast enough for our needs, since the FT publishes or updates articles at most every few minutes, and we cache the generated audio for each article.</p>
<h2 id="but-it-takes-me-an-hour-to-read-my-own-article-and-then-editing-takes-a-few-more-hours">“But it takes me an hour to read my own article, and then editing takes a few more hours!”</h2>
<p>Yes, well, welcome to the future. [We didn’t say this]</p>
<h2 id="how-much-does-it-cost">“How much does it cost?”</h2>
<p>The cost was initially nothing, for the Labs team anyway, because it disappeared into other teams’ AWS budgets, and that is how we roll in Labs. But even when there was proper attribution of costs, eventually, it still turned out to be just pennies per article.</p>
<p>The costs are so low, there are no worries about re-rendering older articles with later versions of the voice as its quality improves incrementally.</p>
<h2 id="it-is-not-very-good-it-gets-lots-of-things-wrong">“It is not very good. It gets lots of things wrong”</h2>
<p>Well, there are at least two angles for this.</p>
<p>No-one can dispute that the new automated voices are amazing, as evidenced by Experimental Geraint doing Bohemian Rhapsody. [Whoops, we did not have that audio to hand.]</p>
<p>The question is then whether they are adequate for long news articles, and there it is a definite maybe. Sometimes definitely no, other times ok.</p>
<p>The artificial voices are particularly bad with whimsy and nuance, and therefore any <a href="https://www.ft.com/robert-shrimsley">Robert Shrimsley article</a>, and often gets the cadence of idioms wrong. The longer the text, the more chances of a blooper bad enough to jolt you out of comfortably listening to the article. Ironically, the better the voices get, the more you relax whilst listening, the worse the jolt when a blooper does happen.</p>
<p>That said, the artificial voices are good enough for isolated paragraphs and sentences.</p>
<h2 id="will-the-voice-quality-improve">“Will the voice quality improve?”</h2>
<p>This is one of the inevitabilities. There is a race to the top among the main providers, competing with each other on voice quality. Experimental Amy had already improved during the course of the project even before we went live. The AWS Polly team (and all the other providers) are actively working on their voice tech.</p>
<p>[Subsequently we’ve had some nice interactions with the AWS team, where they were keen to hear about our hopes and dreams for artificial voices. They were also very polite in asking us why we were spamming their systems with 1000s of spurious requests due to a programming mistake on our part.]</p>
<p>Specific errors are fixed at source by the AWS Polly team, e.g. when we pointed out that the name Trump was being pronounced with the same cadence as the noun trump. Lots of other FT-specific issues are fixed in our own code. Smaller examples include: FT which is pronounced as “ffft” unless we split it into “F.T.”, likewise firstFT, and N.Korea needs to be rewritten as “North Korea” to be read out correctly. Bigger verbal errors arise when site navigation is included in the text, such as when Experimental Amy reads out “click here to see more of firstFT” at the end of an article, or “aitch tee tee pee colon forward slash forward slash” when a url is mentioned verbatim in the text.</p>
<p>We are always finding new gotchas where Experimental Amy just gets things horribly wrong. It is like playing Whack-a-Mole. That said, it’s like operating a ratchet. Each crank of the handle makes the whole thing that bit tighter, and we are finding fewer and fewer gotchas.</p>
<p>It has become very clear that we need to automatically and speedily regenerate the Amy audio after every republish of the article, since people <em>do</em> read along with the voice and it is jarring when they differ.</p>
<p>We tried many variations of the headers and footers of the audio, including the title at the beginning, end, both, or neither. We found that the smaller the intro the better, so we make it clear that this is not the official FT audio, with “This article is narrated by Experimental Amy” at the start, and append the article details such as title and author after the main article content.</p>
<h2 id="what-do-our-readers-think-of-amy">“What do our readers think of Amy?”</h2>
<p>Our primary initial concern is to not antagonise our readers (or Editorial, Lucy) [we looked earnest at this point] and secondly, hopefully, to get some feedback sufficient to justify continuing with this new audio tech. It is not clear if the readers are interested, or have a need for it. Perhaps our subscribers can find a use for it after becoming acclimatised to it. Is the voice good enough for it, or something like it, to become part of the official FT offering?</p>
<p>[It was too soon to tell at the time.
Since then, some strong emotional responses such as</p>
<blockquote>
<p>“i use experimental amy frequently. she is becoming my best friend sadly”</p>
</blockquote>
<p>and some significant evidence of (minority) but persistent and frequent usage.</p>
<p>We have received lots of suggestions pointing out the many details in which it can be improved, often by stripping out spurious text that would not have been narrated if a human had been doing it.</p>
<p>Some new use cases have emerged, beyond basic listening:</p>
<ul>
<li>listening to and reading the same article. This is a thing. Some people like doing both together.</li>
<li>off-tab listening, where the listener can click through to another tab, and the article continues to be narrated out loud</li>
<li>lecture fillers during transitions</li>
<li>the <a href="https://labs.ft.com/2017/02/listen-to-the-ft">obvious use</a> of this automated audio in podcasts</li>
</ul>
<p>]</p>
<h2 id="thank-you">“Thank you”</h2>
<p>We had survived!</p>
<p>After followup emails to check details, there was radio silence for a few days.</p>
<p>Then, on 2017/07/21, Lucy published</p>
<blockquote class="twitter-tweet" data-lang="en">
<p lang="en" dir="ltr"><a href="https://www.ft.com/content/fb705712-3be7-11e7-821a-6027b8a20f23">Amy the robot is no match for me</a>
<br />My AI rival may be cheap and fast but she is dreary
<br />LUCY KELLAWAY</p>
<p><cite>— Financial Times (@FinancialTimes) <a href="https://twitter.com/FinancialTimes/status/866321397496393728">May 21, 2017</a></cite></p>
</blockquote>
<h2 id="-peak-surreality">… Peak Surreality</h2>
<p>Naturally enough, we triggered Experimental Amy to emotionlessly, relentlessly, albeit with jolts, narrate Lucy Kellaway’s article which was (politely) dissing Experimental Amy:</p>
<iframe width="100%" height="100" scrolling="no" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/672209747&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true"></iframe>
<p>And (24 hours later ;-), <a href="https://www.ft.com/content/a0f18048-50b0-4bba-9d59-f4c240dcedf2">Lucy reading Lucy</a>.</p>
<h2 id="geraint-rhapsody">Geraint Rhapsody</h2>
<p>Soon after the chat, we forwarded to Lucy Kellaway a snippet of Experimental Geraint performing Bohemian Rhapsody:</p>
<iframe width="100%" height="100" scrolling="no" frameborder="no" allow="autoplay" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/672208889&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&show_teaser=true&visual=true"></iframe>
<blockquote>
<p>From: Lucy Kellaway <…><br />
Date: Thu, 18 May 2017 12:26:30 +0100<br />
Subject: Re: Audio articles<br />
To: Chris Gathercole <…><br />
<br />
I ADORE GERAINT</p>
</blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]With that, it looked like our foray into artificial voices, might be over. Weeks of work. First day live.BBC newsHack: Multilingual newsrooms2019-07-01T08:30:30+00:002019-07-01T08:30:30+00:00https://labs.ft.com/hackathon/2019/07/01/newshack-translations<p>Last month, we took part in <a href="https://labs.ft.com/hackathon/2018/01/02/bbc-newshack.html" target="_blank">another BBC newsHack</a>, focused on <a href="https://bbcnewslabs.co.uk/2019/06/20/Roundup-of-the-multilingual-newsHACK/" target="_blank">multilingual tools</a>. Having already experimented with <a href="https://labs.ft.com/experiment/2018/11/23/translations.html" target="_blank">automated article translations</a> earlier this year, we thought it might be a good opportunity to expand our exploration of the topic. Along with a cross-disciplinary team from various departments of the FT — and some new acquaintances made on the day — we set out to bring new translations ideas to life.</p>
<blockquote class="twitter-tweet" data-cards="hidden" data-lang="en"><p lang="en" dir="ltr">Four fine folk from from three different <a href="https://twitter.com/FT?ref_src=twsrc%5Etfw">@FT</a> teams have combined to tackle the 'Tools for multilingual newsrooms' <a href="https://twitter.com/hashtag/newshack?src=hash&ref_src=twsrc%5Etfw">#newshack</a> organised by <a href="https://twitter.com/BBC_Connected?ref_src=twsrc%5Etfw">@bbc_connected</a> & <a href="https://twitter.com/BBC_News_Labs?ref_src=twsrc%5Etfw">@BBC_News_Labs</a>: <br><br> <a href="https://twitter.com/Lily_2point0?ref_src=twsrc%5Etfw">@lily_2point0</a>, <a href="https://twitter.com/jamiebrownco?ref_src=twsrc%5Etfw">@jamiebrownco</a>, <a href="https://twitter.com/joannaskao?ref_src=twsrc%5Etfw">@joannaskao</a>, <a href="https://twitter.com/MHA_076?ref_src=twsrc%5Etfw">@MHA_076</a><br><br>(<a href="https://t.co/lUkLbB9O3e">https://t.co/lUkLbB9O3e</a>)</p>— FT Labs (@FTLabs) <a href="https://twitter.com/FTLabs/status/1138388940900225024?ref_src=twsrc%5Etfw">June 11, 2019</a></blockquote>
<h2>Forming an idea</h2>
<figure>
<img src="/assets/uploads/2019/07/newsHack_brainstorm.jpg" alt="Early ideas for the hack on Post-It notes" />
</figure>
<p>Exercises at the beginning of the first day proved really useful to refine our hack. We started with a POINT (Problem, Opportunity, Insight, Need, Theme) system, then formulated ideas starting with “how might we”, carrying on with dot voting and a rapid generation of 8 ideas; to finally settle on one after another round of voting.
</p>
<p>
After some refinement and regrouping of ideas, the team came up with 8 propositions:
<ul>
<li>Connecting readers via voice devices</li>
<li>Translating article comments</li>
<li>Picture drawing about article/concept. “Picture is worth 1000 words”</li>
<li>Highlight text and ask group for translation</li>
<li>Side-by-side thread of a conversation conducted in different languages</li>
<li>Emoji translation -- Universal language?</li>
<li>Linking phrases/idioms that have a very local meaning to a source of explanation automatically</li>
<li>Filtering</li>
</ul>
</p>
<figure>
<img src="/assets/uploads/2019/07/newsHack_ideas.jpg" alt="Sheet of ideas, detailed above" />
</figure>
<p>Fairly soon, propositions started converging towards a core idea: creating a community around languages; an ideal way to break language barriers by discussing and correcting machine translations.</p>
<figure>
<img src="/assets/uploads/2019/07/newsHack_beetle.jpg" alt="The beetle exercise, detailing the usefulness of MetaCrowd" />
</figure>
<h2>MetaCrowd</h2>
<h3>What it is?</h3>
<figure>
<img src="/assets/uploads/2019/07/newsHack_MetaCrowd.png" alt="Comments on an article translation" />
</figure>
<p>MetaCrowd is a commenting system specifically focused on translated content. Poor or unclear translations can be highlighted by readers in articles. The crowd (readers and journalists alike) can respond, comment and discuss, improving readability. A voting feature identifies the most recommended and useful translation for a phrase/keyword, which allows to update the translation for this specific article.</p>
<h3>Demo</h3>
<div class="youtube-container"><iframe class="youtube-iframe" align="center" src="https://www.youtube.com/embed/vArch4SJQ-M?rel=0" frameborder="0" allowfullscreen></iframe></div>
<br>
<h3>Who is it for?</h3>
<p>The tool is primarily for non-native speakers of English who read the news and might use machine translation services to get clarification on sticky points (idioms, mistranslations, etc.)
It also relies on a community of multilingual people/journalists using the same platform to pitch in and answer queries. The main user flow can be seen below:</p>
<figure>
<img src="/assets/uploads/2019/07/newsHack_userflow_1.png" alt="" />
<figcaption>1. French speaker translates English article and seeks clarity on a specific phrase</figcaption>
</figure>
<figure>
<img src="/assets/uploads/2019/07/newsHack_userflow_2.png" alt="" />
<figcaption>2. Native English speaker who is also fluent in French responds to query</figcaption>
</figure>
<figure>
<img src="/assets/uploads/2019/07/newsHack_userflow_3.png" alt="" />
<figcaption>3. French speaker views response to query, and provides a rating</figcaption>
</figure>
<h2>Technical challenges</h2>
<p>Putting a working prototype together in a couple of days came with its challenges. If we had the basic translation framework in place, our efforts were focused on setting up yet another comments system. We are conscious of all the (multilingual) moderations issues that might entail, and for the sake of the prototype, assumed reporting systems were in place. Much more work would be needed in that area for a client-facing product.</p>
<p>Synchronising text highlights proved to be a bit of a hack. Since translations are presented side-by-side with the original text, we thought it would be useful to highlight both sides when text gets selected. We chose text that suited the example; but a word-for-word match is impossible in most cases. There is no cost-effective solution to match each word with their translation, as it would require repeated calls to the translation API, although NLP might be an option?</p>
<figure>
<img src="/assets/uploads/2019/07/newsHack_sync-highlights.png" alt="Demo of synced highlights" />
<figcaption></figcaption>
</figure>
<p>One challenge was with the organisation of the team itself. With participants from outside the FT, we couldn’t share access to our systems, and therefore find the right balance of tasks to share. It seems to have worked out quite well in the end.</p>
<blockquote class="twitter-tweet" data-conversation="none" data-lang="en"><p lang="en" dir="ltr">With a polished presentation which finished *within* the allotted 4 mins, the Metacrowd team included a working demo, and started by getting right to the point... <a href="https://t.co/jqBqhzVozn">pic.twitter.com/jqBqhzVozn</a></p>— FT Labs (@FTLabs) <a href="https://twitter.com/FTLabs/status/1139088075261431808?ref_src=twsrc%5Etfw">June 13, 2019</a></blockquote>
<script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]A further exploration of machine translations at BBC newsHackBarcode2019-02-05T08:30:30+00:002019-02-05T08:30:30+00:00https://labs.ft.com/experiment/2019/02/05/barcode<p><a href="https://github.com/ftlabs/barcode" target="_blank">Barcode</a> is an internal service/endpoint which gathers all of the headline images used in FT articles from a provided date range, the last 24 hours for example, and squashes them to give one condensed image. The final result looks similar to a coloured barcode.</p>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_header_example.jpg"
width="80%"
class=""
/>
<figcaption>Barcode of articles published during 2018-11-13, sorted by colour</figcaption>
</figure>
<h2>How did it come about?</h2>
<p>We were experimenting with ways to show an extremely summarised view of the news. The aim being to give readers a speedier overview of recent events. During the rapid prototyping phase, one idea was to display all of the images from the last 24 hours worth of news all at the same time.</p>
<p>This gave us a large number of images which we tried to display, equally, on the screen at the same time.</p>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_image_grid.png"
width="80%"
class=""
/>
<figcaption>Extreme summarisation headline image grid view</figcaption>
</figure>
<p>Pleasing to look at, however its ability to show a meaningful summary of the news is debatable.</p>
<p>While building the above, and looking for image summarisation inspiration, we came across an image created from every single frame of the Harry Potter movie series.</p>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_hp_films.jpg"
width="80%"
class=""
/>
<figcaption>All the Harry Potter movie frames in one image <a href='https://www.reddit.com/r/dataisbeautiful/comments/194gmp/all_the_harry_potter_frames_from_all_the_movies/' target='_blank'>https://www.reddit.com/r/dataisbeautiful/comments/194gmp/all_the_harry_potter_frames_from_all_the_movies/</a> </figcaption>
</figure>
<p>Even though you cannot see the contents of each individual frame you can see the movies getting darker over each iteration and the (spoilers) white train station scene at the end of the series.</p>
<p>So we made a version of the for the FT news and called it Stretched Images:</p>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_stretched_images.png"
width="80%"
class=""
/>
<figcaption>24 hours of FT headline images displayed using HTML, CSS & JS</figcaption>
</figure>
<p>Similar to the gridded view it doesn't give a great view of what is actually happening in the news. We can, however, see that variety of images used by articles and it does create a pleasing image.</p>
<p>This was as far as the project came under the extreme summarisation banner.</p>
<p>However…we were not done yet.</p>
<h2>Phase 2</h2>
<p>Stretched Images worked on a hardcoded set of parameters and was quite fiddly to update or change. So, in what started as 20% time project, we rebuilt Stretched Images into project Barcode - a parameterised endpoint we could use to explore FT News images.</p>
<p>The first task was to actually turn a stack of images into one single image, as Stretched Images simulated this effect by display a series of stretched images on after the other. So we turned to Node, <a href="https://github.com/lovell/sharp" target="_blank">Sharp</a> & <a href="https://github.com/aheckmann/gm" target="_blank">Graphicsmagick</a>. Node handles the backend image requests and operations, Sharp for resizing and optimisation of images & Graphicsmagick for stitching a series of image files into one image file.</p>
<p>Barcode works in a series of simple steps:</p>
<ul>
<li>
Accept a series of parameters
</li>
<li>
Request images that match the parameters from FT's Origami Image Service
</li>
<li>
Combine all selected images into a single image
</li>
<li>
Save and return that image
</li>
</ul>
<p>We wanted to experiment with the display of the images simpler than Stretched Images so we created a series of parameters to explore barcode images. Some of the initial parameters we added were quite basic:</p>
<ul>
<li>
<b>width</b> - width of final image
</li>
<li>
<b>height</b> - height of final image
</li>
<li>
<b>dateFrom</b> - date to start image selection from
</li>
<li>
<b>dateTo</b> - date to end image selection on
</li>
</ul>
<p>Then we started to come up with some more exploratory features we could mix-n-match for different results:</p>
<ul>
<li>
<b>orientation</b> - stack images horizontally or vertically
</li>
<li>
<b>fit</b> - gets images as masks, squashed or solid colour
</li>
<li>
<b>order</b> - display order of images (published/colour)
</li>
<li>
<b>sort</b> - sort order by asc or desc
</li>
</ul>
<p>We used <a href="https://github.com/sindresorhus/rgb-hex" target="_blank">rgb-hex</a>, <a href="https://github.com/javierbyte/color-sort" target="_blank">Color Sort</a> & <a href="https://github.com/aqrln/image-average-color" target="_blank">Image Average Color</a> to help with the colour conversion, organising and sorting.</p>
<p>Finally we created a service you could query and get some rather interesting images back.</p>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_solid_example.png"
width="80%"
class=""
/>
<figcaption>Horizontal block 'average image colour' lines, sorted by colour</figcaption>
</figure>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_cover_example.png"
width="80%"
class=""
/>
<figcaption>Vertical masked lines, sorted by published date</figcaption>
</figure>
<figure
>
<img alt=""
src="/assets/uploads/2019/01/barcode_fill_example.png"
width="80%"
class=""
/>
<figcaption>Vertical stretched images, sorted by published date</figcaption>
</figure>
<h2>Lessons learned</h2>
<p>A couple of times during the project we planned to implement a feature thinking “this will be quite simple to add”, and quickly realised it was more complex, but not impossible, than we thought.</p>
<ul>
<li>
<b>Creating stretched images</b> - replicating CSS background-position: cover with an image library turned out to be an expensive operation. Thankfully the Origami team gave us a big hand and added an option to the their image service we could use
</li>
<li>
<b>Image generation speeds</b> - - resizing a series of images can be quite process-intensive, we had to try out a few libraries before settling on Sharp
</li>
<li>
<b>Extract pixel colour from an image? Easy...right?</b>- not so much. Thought we could do this without a library, gave up and used Image Average Color insead
</li>
<li>
<b>Sort by colour? Easy...right?</b> - again, not so much. The initial thought was to add up the RGB values and sort by the resulting sum however the result didn't look sorted at all. Color Sort to the rescue
</li>
<li>
<b>Generation timeouts</b> - Some images take longer than average to generate resulting in test server timeouts. We added optimisations to improve the speed of the process
</li>
<li>
<b>Concurrency issues</b> - Our test server refreshes every 24 hours, wiping any generated/cached images. When the server restarts it is vulnerable to a race condition where we could end up with 2 or more items being processed concurrently, as multiple images are requested (with no available cache). We're currently looking into using a mutex flag to restrict access to the queue
</li>
</ul>
<h2>Next steps/future creep</h2>
<p>Throughout the creation of Barcode we were pleasantly plagued by new ideas and features for the service. This is commonly known as feature creep. While quite a few feature creeps made it into the project, there are some that we'd still like to explore:</p>
<ul>
<li>
Adding a greater variety of colour sorting algorithms
</li>
<li>
Add the option to search for images based on other tags, such as genre
</li>
<li>
Use a facial recognition service to identify images of a given person and return a barcode/mosaic of that person
</li>
<li>
Post daily images to a social media thread, "Barcode of the day"
</li>
<li>
Generate an animation (gif or video) of a series of Barcode images, e.g. a month of Barcode images
</li>
</ul>
<p>We've also had a few suggestions from other teams that would be interesting to have a look into.</p>
<p>So perhaps we’re not quite finished with Barcode yet...</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]Creating art out of headline imagesSpeaking in Financial Tongues: Translations on FT.com2018-11-23T08:30:30+00:002018-11-23T08:30:30+00:00https://labs.ft.com/experiment/2018/11/23/translations<p>Researchers have been trying to crack Machine Translation for a few decades. With a wealth of digitised texts, translations APIs are getting quite good; good enough (or so do the native speakers in the office think) to experiment and offer them up to some of our readers. We started out comparing different services, before building a component for ft.com, similar to <a href="https://www.ft.com/content/fb705712-3be7-11e7-821a-6027b8a20f23" target="_blank">Experimental Amy</a>, our text-to-speech component.</p>
<h2>The translations dashboard</h2>
<p>We started by building an internal dashboard, capable of comparing the input in a source language and its translations from various APIs. The dashboard can handle free text, an FT Lexicon reference (an opportunity for laughs with the literal translation of expressions such as “dead cat bounce”), or an FT article ID.</p>
<figure
>
<img alt=""
src="/assets/uploads/2018/11/translations_dash_form.png"
width="50%"
class=""
/>
<figcaption>Options on the translations dashboard</figcaption>
</figure>
<p>Results can be viewed in any available language. With Google offering such a wide range of languages, it is often the case that it cannot be compared with other providers; but it was interesting to see the variations in the output texts for common Indo-European languages.</p>
<p>Beside Google, we also tried DeepL (an algorithm specifically trained on News articles in a handful of languages) and AWS Translate, which was still in its infancy, and lacked the option to preserve paragraphs. For all of them, the quality of translations varies depending on the target language, but is at a satisfying level for European ones.</p>
<figure>
<img src="/assets/uploads/2018/11/translation_dash_results.png" alt="" />
<figcaption>Translations dashboard for <a href="https://ft.com/c4847aca-e7f9-11e8-8a85-04b8afea6ea3" target="_blank">an article</a> translated to French with Google and Deepl</figcaption>
</figure>
<p>If translations APIs are of acceptable quality, they are still costly (with a strange but consistent billing model of charging per amount of characters translated) and required some caching and rate limiting. When requested for the first time, an article translation gets saved into an S3 bucket and will subsequently be served unless the article has been updated since, thus reducing the amount of API calls to the translation service.</p>
<h2>An embedded component for articles</h2>
<p>Having decided on an appropriate translator, we have released the tool to a subset of our readers, on selected articles. A small selection of our readers will now be able to see the translation component (pictured below) on some article pages.</p>
<figure>
<img src="/assets/uploads/2018/11/translation_component.png" alt="" />
<figcaption>Translation component</figcaption>
</figure>
<p>Beyond the selected cohort, several rules apply to the display of the component. We have restricted the initial experiment to <a href="https://www.ft.com/brexit" target="_blank">Brexit</a> and <a href="https://www.ft.com/technology" target="_blank">Technology</a> articles, topics popular with our readers (and especially our readers likely to be in a German-, French-, or Spanish-speaking country, which are our selected languages for now). Secondly, due to the pricing model mentioned above, the component will only display if the monthly target is below the API limit, or if we have a saved translation for the article.</p>
<figure>
<img src="/assets/uploads/2018/11/translation_component_select.png" alt="" />
<figcaption>Translation component interface</figcaption>
</figure>
<p>An option lets users translate all subsequent articles where the component is available in their selected language. Once requested, translations get cached on the client side for the duration of the session, so it is quick to toggle back and forth between English and other languages.</p>
<p>Although satisfying, translations are fully automated and not perfect so it is important that the English version remain “the source of truth” for the article content, as the disclaimer on the component suggests. The API also presents semantic issues; it preserves paragraphs, but not HTML tags and links.</p>
<figure>
<img src="/assets/uploads/2018/11/translation_component_french.png" alt="" />
<figcaption>Translation component with French translation on</figcaption>
</figure>
<p>Translating articles automatically is the first and obvious option, in what could be plethora of other applications if the interest from readers is apparent.</p>
<h2>Further opportunities</h2>
<p>The main application of the translation component is for the comfort of those of our readers whose first language is not English. This might lead to them forwarding the article to relatives or friends who share the same mother tongue. It could also be seen as a learning tool for English speakers who wish to become polyglots.</p>
<p>At the moment, the translations are fully automated, with no correction or human input whatsoever. Moving forward, we could envisage using the dashboard mentioned in the first section to help speed up translation, but save corrections and improvements for each article.</p>
<p>Similarly, for comments, this could allow readers to comment in their native language and see the comment displayed in English for a wider audience to read and engage with.</p>
<figure>
<img src="/assets/uploads/2018/11/translation_gif.gif" alt="Speech bubbles made of newspaper" />
</figure>
<p>Lastly, why limit these options to text? Experimental Amy uses Amazon Polly to generate an audio file of FT articles; Polly supports various voices and dialects, so we could envisage articles being read in French, German, or Spanish from their translation. It could also become a useful tool for Customer Services, transcribing their phone calls into English to be searchable. Or why not help the FT live team with hearables and live speech translation of events? The list of possibilities keeps on growing.</p>
<hr />
<p>Below is a translation to French of the above post, offered without text edits.</p>
<h2>Parler en langues financières : Traductions sur FT.com</h2>
<p>Les chercheurs tentent de casser la traduction automatique depuis quelques décennies. Grâce à la richesse des textes numérisés, les API de traduction sont de plus en plus performantes ; suffisamment bonnes (ou du moins le pensent les locuteurs natifs du bureau) pour les expérimenter et les proposer à certains de nos lecteurs. Nous avons commencé par comparer différents services, avant de construire un composant pour ft.com, similaire à Experimental Amy, notre composant text-to-speech.</p>
<h2>Le tableau de bord des traductions</h2>
<p>Nous avons commencé par construire un tableau de bord interne, capable de comparer l'entrée dans une langue source et ses traductions à partir de différentes API. Le tableau de bord peut gérer du texte libre, une référence au Lexique du CE (une occasion de rire avec la traduction littérale d'expressions telles que "dead cat bounce"), ou un identifiant d'article du CE.</p>
<p>Les résultats peuvent être consultés dans toutes les langues disponibles. Avec Google offrant une telle variété de langues, il est souvent impossible de la comparer avec d'autres fournisseurs, mais il était intéressant de voir les variations dans les textes de sortie pour les langues indo-européennes communes.</p>
<p>En plus de Google, nous avons également essayé DeepL (un algorithme spécifiquement formé sur les articles de nouvelles dans une poignée de langues) et AWS Translate, qui était encore à ses débuts, et n'avait pas la possibilité de préserver les paragraphes. Pour tous, la qualité des traductions varie en fonction de la langue cible, mais se situe à un niveau satisfaisant pour les traductions européennes.</p>
<p>Si les API de traduction sont de qualité acceptable, elles restent coûteuses (avec un modèle de facturation étrange mais cohérent de facturation par nombre de caractères traduits) et nécessitent une certaine mise en cache et une limitation du débit. Lors de la première demande, une traduction d'article est sauvegardée dans un seau S3 et sera ensuite servie à moins que l'article n'ait été mis à jour depuis, ce qui réduit le nombre d'appels API vers le service de traduction.</p>
<h2>Composant incorporé pour articles</h2>
<p>Après avoir choisi un traducteur approprié, nous avons mis l'outil à la disposition d'un sous-ensemble de nos lecteurs, sur des articles sélectionnés. Une petite sélection de nos lecteurs pourra maintenant voir le composant de traduction (illustré ci-dessous) sur certaines pages de l'article.</p>
<p>Au-delà de la cohorte sélectionnée, plusieurs règles s'appliquent à l'affichage de la composante. Nous avons limité l'expérience initiale aux articles Brexit et Technology, sujets populaires auprès de nos lecteurs (et en particulier de nos lecteurs susceptibles d'être dans un pays germanophone, francophone ou hispanophone, qui sont nos langues sélectionnées pour le moment). Deuxièmement, en raison du modèle de prix mentionné ci-dessus, le composant ne s'affichera que si la cible mensuelle est inférieure à la limite de l'API, ou si nous avons une traduction enregistrée pour l'article.</p>
<p>Une option permet aux utilisateurs de traduire tous les articles suivants lorsque le composant est disponible dans la langue de leur choix. Une fois demandées, les traductions sont mises en cache côté client pendant toute la durée de la session, de sorte qu'il est rapide de basculer entre l'anglais et d'autres langues.</p>
<p>Bien que satisfaisantes, les traductions sont entièrement automatisées et ne sont pas parfaites, il est donc important que la version anglaise reste "la source de vérité" pour le contenu de l'article, comme le suggère l'avertissement sur le composant. L'API présente également des problèmes sémantiques ; elle préserve les paragraphes, mais pas les balises et les liens HTML.</p>
<p>La traduction automatique d'articles est la première option évidente, dans ce qui pourrait être une pléthore d'autres applications si l'intérêt des lecteurs est apparent.</p>
<h2>D'autres opportunités</h2>
<p>L'application principale de la composante traduction est pour le confort de nos lecteurs dont la langue maternelle n'est pas l'anglais. Cela pourrait les amener à transmettre l'article à des parents ou amis qui partagent la même langue maternelle. Il pourrait également être considéré comme un outil d'apprentissage pour les anglophones qui souhaitent devenir polyglottes.</p>
<p>Actuellement, les traductions sont entièrement automatisées, sans aucune correction ni intervention humaine. Pour aller de l'avant, nous pourrions envisager d'utiliser le tableau de bord mentionné dans la première section pour aider à accélérer la traduction, mais enregistrer les corrections et améliorations pour chaque article.</p>
<p>De même, pour les commentaires, cela pourrait permettre aux lecteurs de commenter dans leur langue maternelle et de voir le commentaire affiché en anglais pour qu'un public plus large puisse le lire et y participer.</p>
<p>Enfin, pourquoi limiter ces options au texte ? Experimental Amy utilise Amazon Polly pour générer un fichier audio d'articles du CE ; Polly supporte plusieurs voix et dialectes, nous pouvons donc envisager de lire des articles en français, allemand ou espagnol à partir de leur traduction. Il pourrait également devenir un outil utile pour les services à la clientèle, en transcrivant leurs appels téléphoniques en anglais pour être consultable. Ou pourquoi ne pas aider l'équipe en direct du FT avec des auditions et la traduction de discours en direct des événements ? La liste des possibilités ne cesse de s'allonger.</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]Exploring news in different languages.JanetBot: Analysing gender diversity on the FT homepage2018-11-07T08:30:30+00:002018-11-07T08:30:30+00:00https://labs.ft.com/product/2018/11/07/janetbot<p>Every year, the FT hosts an internal hackathon to spark new product ideas. JanetBot’s inception was at the 2017 event, trying to address the ‘Engaging women’ category. It is rare that a hackathon idea finds its way straight into production, but a couple of them did that year. JanetBot is the result of a collaboration between FT Labs, Data Analytics, Editorial Technology, and the FT newsroom (via the Audience Engagement team).</p>
<p>This tool is based on the premise that there are too many “suits” (read “men in suits”) on the FT homepage, according to our customers’ feedback. This is not representative of our readership. Our hypothesis is that female readers might engage more if there are visible reflections of themselves on the page.</p>
<p>The lack of female role models in the media is not an isolated issue, as <a href="https://www.google.com/about/main/gender-equality-films/" target="_blank">this study</a> of women in films by Google and the <a href="https://seejane.org/research-informs-empowers/" target="_blank">Geena Davis Institute on Gender in Media</a> shows. To tackle the problem at our level, we created a tool to run an analysis of gender ratios in the images on the FT homepage.</p>
<h2>Introducing JanetBot</h2>
<p>JanetBot uses Computer Vision to determine which images can be classified as “man”, “woman” or “undefined”. It gives a general classification for the image, rather than individual people represented. So if there is at least one woman in the image, it will get classified as “woman”. “Undefined” is reserved for images where there are no people, or faces are too small or not identifiable.</p>
<p>Images (both main article images and columnists’ headshots) are analysed on our server every 10 minutes and are then distributed in two ways:
<ol>
<li>Results are sent out to a Slack channel in Editorial at scheduled times throughout the day; these times have been picked to match moments when the homepage curation is likely to change, and are in place to minimise disruption.</li>
<figure>
<img src="/assets/uploads/2018/11/janetbot_slack.png" alt="At 07:08 on the UK homepage, it seems the 6 images in the Top Half feature 1 man and 1 woman. It seems the 26 images on the rest of the UK homepage feature 9 men and 7 women. On the INT homepage (...)" />
<figcaption>Slack results</figcaption>
</figure>
<li>At any time, the results are visible on the homepage via an internal plugin. Images are colour-coded with a palette that plays to gender stereotypes to be identified “at a glance”.</li>
<figure>
<img src="/assets/uploads/2018/11/janetbot_display.png" alt="Display of the result son the homepage, with a red border around images of women, blue around images of men, and grey for undefined. An overlay gives the total counts." />
</figure>
</ol>
</p>
<p>The results give the number of images classified as “man”, and the number of images classified as “woman”, out of all the images on the page. There is an emphasis on the first few articles ( what Editorial call the “Top Half”), as JanetBot aims to promote women’s visibility on the page.</p>
<p>The name of the tool itself is an homage to Ms. Yellen, who was featured in no fewer than three articles on the day the bot was first prototyped.</p>
<p>The primary focus since the reporting tool was launched in the newsroom has been to gather long-term data on the curation of our homepage, which will hopefully inform future selection.</p>
<h2>The buzzwords section</h2>
<p>The backend (a Node server) collects images from our content API, which are then classified using Machine Learning and Computer Vision. The original algorithm we used would try to assign a binary gender to any image, regardless of whether it featured people, leading to cars, buildings and logos being 52.345% male or female. When the project moved beyond hackathon stage, we had to implement a way to make the distinction.</p>
<figure>
<img src="/assets/uploads/2018/11/janetbot_cabinet.png" alt="" />
<figcaption>Early run of the model on a picture from cabinet</figcaption>
</figure>
<p>We started our project before Amazon announced their <a href="https://aws.amazon.com/rekognition/" target="_blank">Rekognition</a> service. Yes, other providers were quite good, but were still giving some errors on a small sample. What’s more, we had hopes to train the model on our own corpus of images, making it more “FT”. We soon found that the OpenSource dataset we were running with could not compete with that of Amazon. More on that in a later post.</p>
<p>Amazon Rekognition tags faces with over 95% confidence in most cases, which allows us to note anything below that rate, and discard the results at a later date. Even with a high confidence, the algorithm does not distinguish the main subject of an image from background characters. There could be a woman walking in the background, while President Macron is in focus, and the image would be (rightly, from the algorithm’s point of view) classified as “woman”; whereas JanetBot’s aim is to highlight visible female role models on the page.</p>
<p>Similarly, in the picture below, the result would be “woman” because of the girl identified in the top right corner.</p>
<figure>
<img src="/assets/uploads/2018/11/janetbot_dog.jpg" alt="Picture where the main focus is a dog" />
</figure>
<p>In this other example, the image would be classified as “man” because of the figure on the banknotes.</p>
<figure>
<img src="/assets/uploads/2018/11/janetbot_currency.jpeg" alt="Chinese currency" />
</figure>
<p>It is to avoid these kinds of mistakes that JanetBot also comes with a correction mechanism, where human operators can rectify a classification.</p>
<figure>
<img src="/assets/uploads/2018/11/janetbot_correction.png" alt="" />
</figure>
<p>“What makes a woman?” is a tricky question that has been repeated time and again throughout development. “How do you handle crowds?”: it’s a call for Editorial and the picture desk. The lesson here is that AI is trendy and wonderful, but not everything can be inferred from an image (non-binary genders, for example). Algorithms lack context and human sensitivities and nuances, and that’s not to be forgotten.</p>
<h2>JanetBot, under the hood</h2>
<p>The service is tailored to the <a href="https://ft.com" target="_blank">ft.com</a> homepage which follows a complex set of article streams, rules, and layouts. For example, almost all articles have a main image, which is only displayed according to the layout and its position on the homepage. Whether or not an image is displayed does not come from our content API and has to be worked out by filtering the list of articles present on the page. This part is handled by a Node server which also sends the images to Rekognition and caches the results locally</p>
<p>Those results are then polled by the front page to display the coloured borders, and hooked into Slack (in different teams and channels). There is also a dev channel setup for debugging purposes, that reports errors with various APIs and actions.</p>
<p>Many edge cases became apparent throughout development. Opinion is not always opinion. There are particular cases when columnists’ headshots are displayed. Opinion pieces also occasionally find their way into the Top Half where a headshot could be shown.
Videos and pieces hosted on subdomains had to be catered for.</p>
<p>Another one of our challenges was to handle the various editions of the homepage, which vary depending on a reader’s region, and how to transmit the results to users looking at the page in a different region.</p>
<h2>Towards a diverse future</h2>
<p>After a few months of Machine Learning setbacks, and uncovering/handling edge cases, JanetBot launched in the newsroom to a select group of users. For now, we are saving numbers, analysing, and hopefully informing Editorial’s decisions on article and image curation.</p>
<p>We hope these tools will help the FT offer greater variety in its images, and the stories commissioned. But we are aware it will prove challenging.</p>
<p>Future features might involve a study of diversity beyond gender, although as we’ve seen, when machines are involved, this is a <a href="https://geoffwhite.tech/2018/05/14/facial-recognition-and-ethnicity/" target="_blank">delicate subject</a>. Another possibility would be to have automated suggestions of articles for the front page, particularly if contextual data such as the author, or people mentioned in an article.</p>
<p>Focusing on visual diversity (images) was the most straightforward solution in a hackathon setting and as a first product iteration; but it is only a starting point.</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]From hackathon to MVPPost-it® Poster2018-06-26T08:30:30+00:002018-06-26T08:30:30+00:00https://labs.ft.com/notes/2018/06/26/postit-poster<p>FT Labs have a trade show poster (pictured below) which, for a now-forgotten reason, found itself on display in the office. Very shortly after the poster went up, colleagues started "hacking" it the lo-tech way, with sticky notes.</p>
<img src="/assets/uploads/2018/06/labs_poster.jpg" alt="Trade show poster: FT Labs, Emerging web technology innovation at the Financial Times." />
<blockquote>Emerging web technology innovation at the Financial Times</blockquote>
<p>soon became</p>
<blockquote>Merging techno + innovation at Times</blockquote>
<p>among other slogans.</p>
<p>There is something lovely about the physicality of the poster and notes occasionally falling off, creating new meanings; but we thought we'd spread the fun beyond the sign's location. So we made a <a href="https://labs.ft.com/pages/postit_poster.html">screen version</a> of it, accessible from the comfort of your own computer.</p>
<h2>Digital Post-its</h2>
<p>"Why not have a bit of fun with a "quick" experiment?" is often how things start. We endeavoured to recreate a proportional view of the poster, with the possibility to place notes anywhere, and even to write on them.</p>
<p>It was an opportunity to publicise FT Labs, both externally (with screenshots of the hacked poster) and internally (via our <a href="https://labs.ft.com/2015/09/screens/">screen system</a>).</p>
<p>The best reaction to the Post-its when the project went up on our screens: "Are these real?"; the magic being achieved with a combination of transparency and subtle drop shadows.</p>
<img src="/assets/uploads/2018/06/Poster_v1.jpg" alt="Digital poster: FT Labs, Barging in with technology + innovation for the Financial Times." />
<h2>The techie bit</h2>
<p>There are two parts to the Post-it Poster. First, there is an exact scale replica of the physical sign with which users can interact; then, there's a kiosk view for display only.
<h3>The real deal</h3>
<p>We set ourselves a challenge to keep the digital version as close to the real one as possible, including the gradient, position of the logo and text, and the size of those elements. In practice, that means a lot of percentages, absolute positioning, and viewport units.</p>
<p>The width of the window determines the scroll depth and sets the aspect ratio according to that of the real-life display. As for the notes, they get scaled proportionally to the window size and will keep their position in relation to the text on resize. Double-click on a note to toggle its orientation.</p>
<p>The approach works well when you are able to resize the window or scroll the web page, however that was not an option with our internal screens.</p>
<img src="/assets/uploads/2018/06/Poster_v2.jpg" alt="Digital poster: FT Labs, Emerging at the wrong Time." />
<h3>Kiosk mode</h3>
<p>Internal displays run on Chromebits in kiosk mode. Unfortunately, this means no scrolling and no interaction with the page is possible. It took a bit of head-scratching to translate the Post-its' position to a different screen ratio.</p>
<pre>
<code>//Left positioning
var l = Number(left);
var ratio = (l/100 - 0.1)*window.innerWidth + 70;
//Top positioning
var t = Number(top);
var h = Number(height);
var ratio = 380 + (KIOSK_RATIO*(t - 35.75)/100)*h;</code>
</pre>
<p>These positions are saved from the previous interactive view, and appended as query parameters to the URL.</p>
<pre><code>?type=kiosk&[NOTE_1_PERCENTAGE_TOP];[NOTE_1_PERCENTAGE_LEFT];[NOTE_1_ORIENTATION (h or v)];[NOTE_1_TEXT]-[NOTE_2_PERCENTAGE_TOP];[NOTE_2_PERCENTAGE_LEFT];[NOTE_2_ORIENTATION (h or v)];[NOTE_2_TEXT]</code></pre>
<p>That can make for really long URLs but saves having to maintain a database.</p>
<p>That's it for now, but the list of useless features is potentially endless: how about animating Post-its to fall after a certain amount of time? Or the option to tear or fold the notes? Or an AR version?...</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]A bit of fun with a digital version of our trade show posterYet Another Crossword Data Format2018-03-26T09:30:30+00:002018-03-26T09:30:30+00:00https://labs.ft.com/experiment/2018/03/26/crossword-data-format<p>When starting our <a href="/experiment/2018/03/23/crosswords.html" target="_blank">interactive crossword project</a>, we looked for an existing crossword data format, but not many were documented and widely used. Not wishing to create <a href="https://xkcd.com/927/" target="_blank">Yet Another Standard</a> (thanks, xkcd), we went with <a href="https://www.xwordinfo.com/JSON/" target="_blank">www.xwordinfo.com/JSON</a> but it proved too error-prone, and forced a clumsy parse+build structure on the code. After several acrimonious intra-team discussions, we looked for ‘a better way’, and a new format was born.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s">standard v1</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">Crossword </span><span class="m">15800</span>
<span class="na">author</span><span class="pi">:</span> <span class="s">Julius</span>
<span class="na">editor</span><span class="pi">:</span> <span class="s">Colin Inman</span>
<span class="na">copyright</span><span class="pi">:</span> <span class="s">2018, Financial Times</span>
<span class="na">publisher</span><span class="pi">:</span> <span class="s">Financial Times</span>
<span class="na">pubdate</span><span class="pi">:</span> <span class="s">2018/03/07</span>
<span class="na">size</span><span class="pi">:</span> <span class="s">15x15</span>
<span class="na">across</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">(1,1) 1. Reviewed bistrot/cafe <i>La Maison</i></span><span class="pi">:</span> <span class="s">sandwiches "criminal" (10)</span>
<span class="pi">-</span> <span class="s">(12,1) 6. ,10. Steer clear of wildcat general strike; it's a tragedy eerst (4)</span>
<span class="pi">-</span> <span class="s">(1,3) 9. New salary caps not introduced at work; it gets up my nose (5,5)</span>
<span class="pi">-</span> <span class="s">(12,3) 10. See 6 across (4)</span>
<span class="pi">-</span> <span class="s">(4,5) 12. Live like a king with dad in castles in far-flung places (7,5)</span>
<span class="pi">-</span> <span class="s">(1,7) 15. Got up and prepared the pizza base (6,3)</span>
<span class="pi">-</span> <span class="s">(11,7) 17. She's an inspiration for occasionally weird author (5)</span>
<span class="pi">-</span> <span class="s">(1,9) 18. Heads of Foreign Office committee unusually struggle to concentrate . . . . (5)</span>
<span class="pi">-</span> <span class="s">(7,9) 19. . . . . as multiple channels stream Tory with excellent oratory skills (9)</span>
<span class="pi">-</span> <span class="s">(1,11) 20. Design squalid sites as means to raise quick cash? (6,6)</span>
<span class="pi">-</span> <span class="s">(1,13) 24. ,26. Suites arranged to suit Norwegian's booking? (4)</span>
<span class="pi">-</span> <span class="s">(6,13) 25. Like the tale of bird fencing Romeo's mate's stolen car? (10)</span>
<span class="pi">-</span> <span class="s">(1,15) 26. See 24 across (4)</span>
<span class="pi">-</span> <span class="s">(6,15) 27. Fake uni degrees heavily attacked (5,5)</span>
<span class="na">down</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">(1,1) 1. ,2. Somewhat hormonal, Isabella is an enigmatic figure (4)</span>
<span class="pi">-</span> <span class="s">(3,1) 2. See 1 down (4)</span>
<span class="pi">-</span> <span class="s">(5,1) 3. Didn't Trump go after journalist with court case? (8,4)</span>
<span class="pi">-</span> <span class="s">(7,1) 4. Old Bill gathered wood (5)</span>
<span class="pi">-</span> <span class="s">(9,1) 5. Hirsute character in Oxford University called Brown (9)</span>
<span class="pi">-</span> <span class="s">(13,1) 7. In European empty bar I had dinner date resembling Edna (10)</span>
<span class="pi">-</span> <span class="s">(15,1) 8. Rustic logs chopped for those making camp fire? (4,6)</span>
<span class="pi">-</span> <span class="s">(11,4) 11. Names US mountains as stated by Yankee (12)</span>
<span class="pi">-</span> <span class="s">(1,6) 13. Let rip guff; foul smell gets up one's nose! (7-3)</span>
<span class="pi">-</span> <span class="s">(3,6) 14. Piece of ebony ruler? (5,5)</span>
<span class="pi">-</span> <span class="s">(7,7) 16. Not for the first time, falls over, leaving artist to support old morthern church (4,5)</span>
<span class="pi">-</span> <span class="s">(9,11) 21. Composer putting note in envelope (no stamp required) (5)</span>
<span class="pi">-</span> <span class="s">(13,12) 22. ,23. Book a month to take over from glum donkey (4)</span>
<span class="pi">-</span> <span class="s">(15,12) 23. See 22 down (4)</span>
</code></pre></div></div>
<p>The above example of the new crossword data format describes <a href="https://labs.ft.com/crosswords/2018/03/07/Crossword-15800.html" target="_blank">Crossword No.15,800 Set by Julius on 2018/03/07</a>,</p>
<figure>
<img alt="A example of the new crossword data format" src="https://www.ft.com/__origami/service/image/v2/images/raw/ftcms:bdb7ea5a-1875-11e8-9e9c-25c814761640?source=crosswordsftcom" width="100%" class="aligncenter size-full" />
</figure>
<p>and drives <a href="https://labs.ft.com/crosswords/2018/03/07/Puzzle-15800-dynamic.html" target="_blank">an interactive version</a>.</p>
<h2 id="the-new-format">The new format</h2>
<p>The crossword data format is a DSL, <a href="https://en.wikipedia.org/wiki/Domain-specific_language" target="_blank">a Domain Specific Language</a>, being an concise, clear, human-readable piece of text that specifies how to construct a simple crossword.</p>
<p>It is written in <a href="https://en.wikipedia.org/wiki/YAML" target="_blank">YAML</a>. Or rather, nearly YAML. The line items for each clue need to be wrapped in quotes in order to accommodate colons and speech marks in the clue. NB, currently our implementation fails to take this into account, parsing the text directly and not as YAML.</p>
<p>Blank lines and comments (starting with #) are ignored.</p>
<p>There are several mandatory fields such as ‘name’ (which should be unique across all the crosswords), ‘author’, and ‘size’.</p>
<h3 id="clues">Clues</h3>
<p>The clues are grouped as lists named ‘across’ and ‘down’.</p>
<p>Each clue has the form: - <code class="language-plaintext highlighter-rouge">(COORDINATES) IDs. Clue text (ANSWER)</code></p>
<p>COORDINATES are (across, down), with (1,1) being the top left square.</p>
<p>The IDs identify each clue, and also tie together subsequent (comma-separated) clue IDs for answers which straddle multiple positions on the grid, e.g., the clue ‘6 across‘ in the example grid above. The subsequent IDs numbers might have an associated ‘down’ or ‘across’ if needed to disambiguate when a particular clue id has both aspects.</p>
<p>The ANSWER is a sequence of words or numbers separated by commas or hyphens. Very obscure crosswords might want to have full stops, colons, or semi-colons as separators in there too, and this format could handle them.</p>
<p>The Clue text can include basic html formatting such as italics.</p>
<h3 id="weaknesses">Weaknesses</h3>
<p>The format seems flexible enough to cover all the basics for the standard crossword, but is likely to run into difficulties for non-standard variants.</p>
<p>If we stick with official YAML, then we need to take extra care about escaping certain characters used in the clues. There can be no bare square brackets, speech marks, colons, etc in the clue and answer texts, but the line (after the initial hyphen) can be wrapped in quotes to overcome that.</p>
<p><em>Breaking News!</em> It turns out that some crossword authors have been known to split individual answer words across multiple lines in the grid! Yes. Shocking. This did not figure even as a possibility during the initial design of the format. I’m assured this only happens with words which break naturally into sub words, e.g. LIGHTHOUSE -> LIGHT HOUSE, when it might have been tiresome to have too many clues with short answers. Here is <a href="https://labs.ft.com/crosswords/2018/03/24/Crossword-15815.html">a recent example</a> (look for clues ‘22 across’ and ‘7 down’).</p>
<p>The new format, as it stands, cannot quite handle that, and it had to be <a href="https://labs.ft.com/crosswords/2018/03/24/Puzzle-15815-dynamic.html">fudged a bit</a>. However, the format could be fairly easily and cleanly extended to accommodate that scenario, perhaps by using a different separator in the IDs, e.g. for 22 across, using a pipe character to indicate the answer which spans clues 22 and 27 is a single word of length 10 (4+6):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- (12,11) 22.|27. Acrobatics in toilets with Eliot's initials <i>[NB: one word, length 10]</i> (4)
...
- (1,15) 27. See 22 across (6)
</code></pre></div></div>
<h2 id="using-the-format">Using the format</h2>
<p>The format is easy to read, debug, and edit manually. This is probably its main benefit when compared with the other formats out there.</p>
<p>It is easy to parse and validate, at a basic level, to check that all the mandatory details and structure are present and correct.</p>
<p>A little more effort (by unpacking the clues into a grid) is needed to ensure that</p>
<ul>
<li>the answers fit within the stated size of the grid,</li>
<li>the clues are numbered appropriately,</li>
<li>and the answer letters agree at every intersection .</li>
</ul>
<p>We have several different incoming data formats for crosswords published by the FT, including some (undocumented) XML exported from a commercial crossword compiling tool, <a href="https://www.crossword-compiler.com/" target="_blank">CCW</a>, and a bespoke (but consistently-formatted) Word document. It has proved relatively simple to transform them into the new YAML format, and import that into our new crossword implementation.</p>
<h2 id="implementation--infrastructure">Implementation / Infrastructure</h2>
<p>A couple of bugs/oversights have become apparent in our definition and implementation of this format, but let’s pretend they are not a big issue for now.</p>
<p>We have an in-browser <a href="https://labs.ft.com/pages/crossword_dsl.html" target="_blank">parser / validator / translator / editor / viewer</a> which makes it easy to import and edit crosswords from a variety of sources. The code is … horrible; a classic example of a hack which stayed around too long. The main fiddliness is in handling the multi-clue answers.</p>
<p>The same parser is baked into an <a href="https://origami.ft.com/" target="_blank">Origami</a> web <a href="https://registry.origami.ft.com/components/o-crossword@1.8.2" target="_blank">component</a> which is used to power the current online test. The crossword data can be supplied as a url to a file, or directly as an attribute value.</p>
<p>We use <a href="https://jekyllrb.com/" target="_blank">Jekyll</a> to create the mini site for the crossword experiment, <a href="https://labs.ft.com/crosswords" target="_blank">labs.ft.com/crosswords</a>.</p>
<h2 id="next-steps--extensions">Next steps / Extensions</h2>
<p>The next, obvious step is to fix the YAML-related format issues uncovered during the writing of this blog post.</p>
<p>The parser needs a rewrite, and some unit tests would be nice.</p>
<p>The format can be extended in a variety of straightforward ways but, since complexity was one of the negative aspects of the other formats, that is to be resisted.</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]Defining a new human-readable data format for simple crosswordsInteractive crossword2018-03-23T08:30:30+00:002018-03-23T08:30:30+00:00https://labs.ft.com/experiment/2018/03/23/crosswords<p>Crosswords have been around for over a hundred years, and the source of vehement opinions from the very start. When computers came along, it would only be a matter of time before crosswords joined the pool of digital pastimes, up there with Minesweeper or Spider Solitaire.</p>
<p>Online, there is almost a duty for news publishers to offer an interactive version of their games page. There is a push towards interactive (admittedly of varying quality). Crosswords are important to some readers and are even the main feature of one of the New York Times' subscription tiers (What a turnaround when you know that, in 1924, they deemed it a <cite>"sinful waste in the utterly futile finding of words" * </cite>!) *Source: <a href="https://en.wikipedia.org/wiki/Crossword#History" target="_blank">Crossword - Wikipedia</a></p>
<p>FT Labs have been working on a <a href="https://registry.origami.ft.com/components/o-crossword" target="_blank">custom interactive crossword component</a> with all the features you would expect and more, giving a particular focus to the clue list, for a better cross-device experience and improved overall accessibility.</p>
<figure>
<img src="/assets/uploads/2018/03/FT_crossword.png" alt="" />
</figure>
<h2>Making cruciverbalists happy</h2>
<blockquote>
<p>It was always astonishing to me as a newspaper editor how much readers cared about their puzzles. Make a mistake, leave them out of the paper for a day, and the telephone wouldn’t stop ringing.<p>
<p><cite>John Temple - <a href="http://www.niemanlab.org/2017/06/solving-the-crossword-puzzle-rebuilding-a-print-habit-on-digital-devices/" target="_blank">Nieman Lab</a></cite></p>
</blockquote>
<p>We concur with the above; the feeling for crosswords seems to be universal. Ever since receiving complaints for the lack of crossword on the site, the FT has provided PDF versions of its daily grid.</p>
<p>It has been a long time coming, but we now have an interactive version too. It comes with all the usual features: selecting the clue and highlighting the relevant position in the grid, selecting a cell and highlighting the corresponding clue, saving the answers of a crossword in progress. (Note, however, that we are not validating the answers, as the FT crossword involves prizes and the answers are published at a later date).</p>
<p>And, to make up for the delay, this new interactive crossword comes with additional features that we think cruciverbalists might find useful and/or interesting.</p>
<h2>Novelty for all</h2>
<h3>A cross-platform feature: the list view</h3>
<!-- Flash -->
<p>If some crossword apps are still built in Flash, in the day of responsive web apps, that simply wouldn't cut it for us. The real head-scratcher with this experiment was to find a way to create an effective and enjoyable experience on all kinds of devices.</p>
<p>More than on the crossword grid alone, we felt the area for experimentation and innovation must be the clue list itself. Focusing on the list, one of our first explorations was to create a minimised list, with a "scroll-and-zoom" effect to reaveal the clues. Otherwise, a clue is only revealed when a word location is selected in the grid. The "Force-Compact" demo is available <a href="https://www.ft.com/__origami/service/build/v2/demos/o-crossword@0.2.0/force-compact" target="_blank">online</a> and demonstrated below.</p>
<p>
<div class="youtube-container"><iframe class="youtube-iframe" align="center" src="https://www.youtube.com/embed/PUAPFeQTvBc?rel=0" frameborder="0" allowfullscreen></iframe></div>
</p>
<p>Another iteration, and its current form, moves the interaction away from the grid. Shifting the focus to the clue list by giving the possibility to answer within it has several advantages. On mobile devices, where the grid cells would be too small to select, the list is more practical to navigate and interact with. Even in the grid view, the answer should be given in the input above, which highlights the selected clue. Whichever type input you choose, grid or list, on mobile or desktop, partial answers in both areas will be updated in real time.</p>
<h4>Mobile view demo</h4>
<p>
<div class="youtube-container"><iframe class="youtube-iframe" align="center" src="https://www.youtube.com/embed/_9GDGBFlvG4?rel=0" frameborder="0" allowfullscreen></iframe></div>
</p>
<p>From an accessibility standpoint, it is also a benefit.</p>
<h3>Accessibility</h3>
<p>This new feature ensures the accessibility of the crossword, where keyboard navigation and interaction focus on the list itself without the need for the grid (but still there as an alternative visual display).</p>
<p>We also took care to add extra information for screen readers:</p>
<ul>
<li>When a clue in the list is selected, it is read out and instructions on how to input an answer are given.</li>
<li>Completed or partially completed answers are read out affter the clue</li>
<li>Consecutive letters left blank are grouped. A screen reader would therefore say: <cite>"A 3 blanks T E"</cite> instead of <cite>"A blank blank blank T E"</cite>.</li>
</ul>
<h3>Print</h3>
<p>Being conscious that we are coming from a printed grid and currently offering up PDFs to print out, we took particular care with the print view of our new interactive experience.</p>
<p>So, for those in fear of missing the feel of paper, the grid and its clues can be printed directly from the browser, either empty or partially filled in, in portrait and landscape, for those times when a pencil is more practical than a keyboard.</p>
<figure>
<img src="/assets/uploads/2018/03/Crossword_print_preview.png" alt="" />
<figcaption>Print preview in the browser</figcaption>
</figure>
<h2>Technical challenges</h2>
<h3>The complexity of new features</h3>
<p>One of the biggest challenges with the new feature was to synchronise inputs across the grid and the list view, and figuring out which was each letter's rightful place. This was achieved by relying heavily on <code>data-</code> attributes.</p>
<p>Similarly, with our focus on accessibility, ensuring content that was to be read out by screen readers was up-to-date proved arduous.</p>
<h3>Working across devices</h3>
<p>Guaranteeing the crossword would work across devices, and particularly on mobile phones also came with its fair share of questions and obstacles.</p>
<p>Firstly, we knew that the crossword grid could either be 15x15 or 17x17 cells. How would we render this on a very small screen (below 400px in width)? The solution was to make the most of our newly implemented list view, and add the selected clue (and answer inputs) above the grid; which means interaction with the grid on mobile isn't necessary anymore, but it is there, visually.</p>
<figure>
<img src="/assets/uploads/2018/03/mobile_views.png" alt="" />
<figcaption>The grid and list views on mobile</figcaption>
</figure>
<p>An <a href="https://registry.origami.ft.com/components/o-crossword@0.2.0" target="_blank">earlier version</a> of the experiment envisaged having a desktop-size grid scrolling horizontally.</p>
<p>Secondly, we found that various mobile platforms handled keyboard input and focus differently. For example, on Android devices, trying to override input content would result in printing `Å`, no matter which letter of the keyboard was selected.</p>
<p>Finally, we implemented a way to save progress for each interactive crossword. However, on mobile devices, this will be limited by the amount of local storage. Since we are not authenticating users, there is no way to save this across devices and store the answers against their FT account.</p>
<h3>Generating new grids</h3>
<p>For now, the process to publish the interactive versions of the daily crossword is partly manual. Obviously, a fully automated publication would be ideal, but the main steps are automated. There will be a subsequent post describing the new crossword data format and the tool for translating between formats.</p>
<h2>Available now!</h2>
<p>Please try the latest interactive crosswords at <a href="https://labs.ft.com/crosswords/" target="_blank">labs.ft.com/crosswords/</a> and let us know what you think.</p>
<h2>Addendum</h2>
<p>For more detail about the underlying crossword data format used in this experiment, see the <a href="/experiment/2018/03/26/crossword-data-format.html" target="_blank">following post</a></p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]Bringing an interactive crossword to FT.comTech Radar2018-02-20T08:30:30+00:002018-02-20T08:30:30+00:00https://labs.ft.com/2018/02/20/tech-radar<div class="embedded half-right clearfix">
<div class="embed-text">
<p class="firstpar">FT Labs’ Tech Radar is a highly configurable system for auto-generating ‘radar’-style displays of project and team datasets such as backlogs.</p>
<p class="secondpar">The code sits <a href="https://github.com/ftlabs/tech-radar">in this repo</a>, and includes <a href="https://raw.githubusercontent.com/ftlabs/tech-radar/master/client/demo.json">an example dataset</a> which is the source of data for <a href="https://labs.ft.com/tech-radar/?json=https://labs.ft.com/tech-radar/demo.json">this fruit and veg example</a>.</p>
</div>
<figure style="float: right;">
<img alt="" src="/assets/uploads/2018/02/example_radar.png" width="300" class="" />
<figcaption>Fruit & Veg Demo</figcaption>
</figure>
</div>
<h2 id="why-a-radar-and-why-now-then">Why a radar, and why <del>now</del> then?</h2>
<p>It seemed every internal team wanted their own radar-themed view of their projects, and were grumbling about the effort needed to maintain them manually. Greatly inspired by the <a href="https://www.thoughtworks.com/radar">Thoughtworks Radar</a>, we saw an opportunity to build a tool to automate the generation of radar displays, run and configured in-browser, taking advantage of an in-house service which exports data from Google Spreadsheets as a JSON feed. Each team could maintain (and share) their projects’ details in a spreadsheet, and launch the radar display via a link within the spreadsheet.</p>
<h2 id="how">How?</h2>
<p>The FT Labs Tech radar tool is built using Javascript using the <a href="https://d3js.org/">D3 library</a>’s force diagram, served as a static page from <a href="http://labs.ft.com/tech-radar/">Github Pages</a>. The radar is configurable via a display dashboard, via url parameters (including one to hide the dashboard…), and via parameters embedded in the JSON data.</p>
<figure>
<img alt="" src="/assets/uploads/2018/02/partial_dashboard.png" width="100%" class="" />
<figcaption>Part of the dashboard, for the Display tab</figcaption>
</figure>
<p>Nodes in the display are (initially) placed using a combination of attraction (to the section they should be in) and repulsion (from each other). Labels attempt to avoid all other labels and nodes, whilst still keeping close to their own node, in order to remain visible and relevant.</p>
<h2 id="usage">Usage</h2>
<p>Configure the radar url with a source of JSON data, e.g. the handy demo data file,</p>
<p><a href="https://labs.ft.com/tech-radar/?json=https://labs.ft.com/tech-radar/demo.json">https://labs.ft.com/tech-radar/?<em>json=https://labs.ft.com/tech-radar/demo.json</em></a></p>
<p>The data file must have a field called ‘name’. If there are any configs specified in the file, they are set via the configvalue field, which is null otherwise. In the basic demo, it specifies the ‘rainbow’ colour scheme for the rings <code>{"name": "ringcol", "configvalue": "rainbow"}</code>, and the field called ‘tastiness’ as the default data field <code>{"name": "sortcol", "configvalue": "tastiness"}</code>. The other data fields will be available in the dashboard for configuring the display.</p>
<p>Within the FT, create (or update) a simple dataset with named columns in a google spreadsheet, export it as a JSON feed using the in-house service, and configure the radar url with the name and id of that entry. See <a href="https://github.com/ft-interactive/bertha">the Bertha repo</a>.</p>
<p>When viewing the radar display, all the parameters are adjustable directly in the dashboard, in turn updating the url of the radar page. The url can be shared and the recipient will get the same display.</p>
<p>After the display has rendered, with the nodes and labels positioned automatically, they can be repositioned individually. The user will need to take a screenshot to accurately preserve such a manually adjusted layout, since those manual adjustments are not persisted in the url.</p>
<div class="row clearfix">
<figure style="float: left;">
<img alt="" src="/assets/uploads/2018/02/positioning_default.png" width="300" class="" />
<figcaption>Default positions, for the display tab</figcaption>
</figure>
<figure>
<img alt="" src="/assets/uploads/2018/02/positioning_manual.png" width="300" class="" />
<figcaption>Manual positioning</figcaption>
</figure>
</div>
<p>In the display, an item can be expanded in the table, and both the dashboard and the table can be removed from the display completely.</p>
<h2 id="features-and-requirements">Features and requirements</h2>
<p>We had some noble users who suffered through the early versions and helped us shape the final version.</p>
<p>Their main requirements were:</p>
<ul>
<li>Every node should be in the correct sector</li>
<li>Choosing their own colour scheme</li>
<li>Being able to manually adjust the nodes and labels</li>
<li>And, somewhat unexpectedly, being able to vary the origin of the quadrant</li>
</ul>
<p>The default settings should just work (™), but it is possible to override almost all aspects of the display, including:</p>
<ul>
<li>The balance of repulsion vs attraction</li>
<li>Colour(s), colour gradient, and crystallisation row (where the colour brightness peaks at one of the middle rows, rather than an end row)</li>
<li>Which datasource(s). It is possible to combine multiple data sources into one display.</li>
<li>Which non-default columns to use as the display dimensions, and their sort order</li>
<li>Filtering the data</li>
<li>Sub-dividing the data into sectors (using a second sort field defined by the param ‘segment’)</li>
<li>Which corner is the origin of the radar quadrant</li>
<li>Any of the CSS</li>
<li>And lots more. <a href="https://github.com/ftlabs/tech-radar/blob/master/README.md">See the repo README for details</a>.</li>
</ul>
<p>And, when all is said and done, every node, node label, and name can be individually repositioned and locked into place, if their default placements are not satisfactory.</p>
<div class="quadrant clearfix">
<figure style="float: left;">
<img alt="" src="/assets/uploads/2018/02/quadrant_lower_right.png" width="300" class="" />
<figcaption>Config: ringcolor: rainbow</figcaption>
</figure>
<figure style="float: left;">
<img alt="" src="/assets/uploads/2018/02/quadrant_lower_left.png" width="300" class="" />
<figcaption>Config: ringcolor: green</figcaption>
</figure>
<figure style="float: left;">
<img alt="" src="/assets/uploads/2018/02/quadrant_upper_right.png" width="300" class="" />
<figcaption>Config: ringcolor: yellow, crystallisation: 0</figcaption>
</figure>
<figure style="float: left;">
<img alt="" src="/assets/uploads/2018/02/quadrant_upper_left.png" width="300" class="" />
<figcaption>Config: ringcolor: red, crystallisation: 3</figcaption>
</figure>
</div>
<h2 id="difficulties">Difficulties</h2>
<h3 id="physics">Physics</h3>
<ul>
<li>Trying to stay within bounded regions using attraction+repulsion. We had to resort to a bit of maths at the boundaries and forcibly repatriate run-away nodes into the quadrant. This can easily happen even when the region is not particularly crowded.</li>
<li>Fitting lots of nodes into tight corners. As has no doubt been a problem for everyone who tries to build radar-themed displays, space gets tight near the origin, especially when specifying a segment field.</li>
<li>Keeping the labels from overlapping. We used invisible nodes in D3 to repel labels from all other labels and nodes (except their own node).</li>
</ul>
<h3 id="ux">UX</h3>
<ul>
<li>Juggling a large set of parameters to find a combination that worked well in most circumstances. For example, <a href="https://labs.ft.com/tech-radar/?json=https%3A%2F%2Flabs.ft.com%2Ftech-radar%2Fdemo.json&sortcol=tastiness&segment=colour&sorttype=alphabetical&title=FTLabs+Potential+Projects+State&ringcolor=rainbow&proportionalrings=true&noderepulsion=1&nodeattraction=9&quadrant=bottom+left"">this more fancy example</a>.</li>
<li>Not succumbing to feature creep <em>(News Update: we succumbed)</em>.</li>
<li>Handling the explosion in config parameters, many of which were purely experimental, and trying to avoid the configuration becoming a UX disaster. Designing a useful, usable dashboard was a significant challenge.</li>
</ul>
<figure>
<img alt="" src="/assets/uploads/2018/02/fancy_sectors.png" width="80%" class="" />
<figcaption>'More Fancy' Config: segment: color, noderepulsion: 1, nodeattraction: 9, proportionalrings: true</figcaption>
</figure>
<h3 id="browsers">Browsers</h3>
<ul>
<li>Cross-domain issues. The JSON file needs to come from the same domain.</li>
<li>HTTPS. The JSON file url needs to be HTTPS if the radar page is on HTTPS.</li>
</ul>
<h2 id="whats-next">What’s Next?</h2>
<p>Despite there already being far too many configurable features, the urge is obviously to continue to add more. We won’t be doing so, unless internal users clamour for it. Meanwhile, this code is open source, so…</p>[#<Jekyll::Document _author/adam.md collection=author>, #<Jekyll::Document _author/andrew.md collection=author>, #<Jekyll::Document _author/ftlabs.md collection=author>, #<Jekyll::Document _author/index.md collection=author>, #<Jekyll::Document _author/matthewandrews.md collection=author>, #<Jekyll::Document _author/robert.md collection=author>]A highly configurable system for auto-generating ‘radar’-style displays of project and team datasets such as backlogs.