Posts tagged ‘research’
The government’s zealous War on Hackers threatens us, the good hackers who stop the bad ones. They can’t tell the good witches from the bad witches. When members of our community get threatened by the system, we should probably do more to stand in solidarity with them. I mention this because many of you will be flying to SFO this coming week for the RSA Conference, which gives us an opportunity to show solidarity.
Today, a security researcher tweeted a joke while on a plane. When he landed, the FBI grabbed him and confiscated all his stuff. The tweets are here:
Find myself on a 737/800, lets see Box-IFE-ICE-SATCOM, ? Shall we start playing with EICAS messages? “PASS OXYGEN ON” Anyone ? :)
— Chris Roberts (@Sidragon1) April 15, 2015
Chris Roberts’ area of research is embedded control systems like those on planes. It’s not simply that the FBI grabbed him because of a random person on a plane, but specifically because he’s a security researcher. He’s on the FBI’s radar (so to speak) for things like this Fox News interview.
I suggest we all start joke tweeting along these lines, from the airplanes, like:
DFW->SFO. Playing with airplane wifi. I hope the pilots enjoy the Rick Astely video playing on their EICAS system.
LGA->SFO. Note to self. Don’t fuzz the SATCOM unit while on Twitter. Takes GoGo an hour to come back up.
NRT->SFO. Yup, the IFE will grab corrupt MP3 from my iPhone and give a shell. I wonder if nmap will run on it.
PDX->SFO. HackRF says there’s a strong 915 MHz qpsk 64k symbol/second signal. I wonder what’ll happen if I replay it.
The trick is to write jokes, not to actually threaten anything — like the original tweet above. Those of us with technical knowledge and skills should be free to express our humor without the FBI confiscating all our stuff when we land.
Bye bye electronics, all encrypted….and all now in custody/seized pic.twitter.com/a5o6rYTbZ0
— Chris Roberts (@Sidragon1) April 16, 2015
BTW, I know you can all steal in-flight WiFi easier than you can pay for it, but do pay for it :)
Overview of the April 2015 Microsoft patches and their status.
|#||Affected||Contra Indications – KB||Known Exploits||Microsoft rating(**)||ISC rating(*)|
|MS15-032||Cumulative Security Update for Internet Explorer
|CVE-2015-1652, CVE-2015-1657, CVE-2015-1659, CVE-2015-1660, CVE-2015-1661, CVE-2015-1662, CVE-2015-1665, CVE-2015-1666, CVE-2015-1667, CVE-2015-1668||KB 3038314||No||Severity:Critical
|MS15-033||Vulnerabilities in Microsoft Office Could Allow Remote Code Execution
(ReplacesMS14-081 MS15-022 )
|KB 3048019||vuln. public.||Severity:Critical
|MS15-034||Vulnerability in HTTP.sys Could Allow Remote Code Execution|
|MS15-035||Vulnerability in Microsoft Graphics Component Could Allow Remote Code Execution|
|MS15-036||Vulnerabilities in Microsoft SharePoint Server Could Allow Elevation of Privilege
|MS15-037||Vulnerability in Windows Task Scheduler Could Allow Elevation of Privilege|
|MS15-038||Vulnerabilities in Microsoft Windows Could Allow Elevation of Privilege
(ReplacesMS15-025 MS15-031 )
|MS15-039||Vulnerability in XML Core Services Could Allow Security Feature Bypass
|MS15-040||Vulnerability in Active Directory Federation Services Could Allow Information Disclosure|
|MS15-041||Vulnerability in .NET Framework Could Allow Information Disclosure
|MS15-042||Vulnerability in Windows Hyper-V Could Allow Denial of Service|
We appreciate updates
US based customers can call Microsoft for free patch related support on 1-866-PCSAFETY
- We use 4 levels:
- PATCH NOW: Typically used where we see immediate danger of exploitation. Typical environments will want to deploy these patches ASAP. Workarounds are typically not accepted by users or are not possible. This rating is often used when typical deployments make it vulnerable and exploits are being used or easy to obtain or make.
- Critical: Anything that needs little to become interesting”>Less Urt practices for servers such as not using outlook, MSIE, word etc. to do traditional office or leisure work.
- The rating is not a risk analysis as such. It is a rating of importance of the vulnerability and the perceived or even predicted threatatches.
Alex Stanford – GIAC GWEB GSEC,
Research Operations Manager,
SANS Internet Storm Center
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
In recent years blockades of “pirate” websites have spread across Europe and elsewhere. In the UK, for example, more than 100 websites are currently blocked by the major ISPs.
Opponents of this censorship route often argue that the measures are ineffective, and that people simply move to other sites. However, in its latest Digital Music Report music industry group IFPI disagrees, pointing at research conducted in the UK.
“Website blocking has proved effective where applied,” IFPI writes, noting that the number of UK visits to “all BitTorrent” sites dropped from 20 million in April 2012 to 11 million two years later.
The key to an effective blocking strategy is to target not just one, but all leading pirate sites.
“While blocking an individual site does not have a significant impact on overall traffic to unlicensed services, once a number of leading sites are
blocked then there is a major impact,” IFPI argues.
For now, however, courts have shown to be among the biggest hurdles. It can sometimes take years before these cases reach a conclusion, and the same requests have to be made in all countries.
To streamline the process, copyright holders now want blocking injunctions to apply across borders, starting in the European Union.
“The recording industry continues to call for website blocking legislation where it does not already exist. In countries where there is already a legal basis for blocking, procedures can be slow and burdensome,” IFPI writes.
“For example, within the EU, blocking The Pirate Bay has meant taking multiple legal actions in different member states and rights holders are calling for injunctions to have cross-border effect.”
In addition to website blockades the music industry also stresses that other stakeholders should do more to help fight piracy. Search engines should prioritize legal services, for example, and advertisers and payment processors should cut their ties with pirate sites.
While IFPI’s numbers suggests that BitTorrent piracy has decreased globally, it still remains a significant problem. The group estimates that there are still four billion pirated music downloads per year on BitTorrent alone.
In other words, there’s plenty of blocking to be done before it’s no longer an issue, if that point will ever be reached.
China has been actively diverting unencrypted Web traffic destined for its top online search service — Baidu.com — so that some visitors from outside of the country were unwittingly enlisted in a novel and unsettling series of denial-of-service attacks aimed at sidelining sites that distribute anti-censorship tools, according to research released this week.
The findings, published in a joint paper today by researchers with University of Toronto’s Citizen LabCitizen Lab, the International Computer Science Institute (ICSI) and the University of California, Berkeley, track a remarkable development in China’s increasingly public display of its evolving cyber warfare prowess.
“Their willingness to be so public mystifies me,” said Nicholas Weaver, a researcher at the ICSI who helped dig through the clues about the mysterious attack. “But it does appear to be a very public statement about their capabilities.”
Earlier this month, Github — an open-source code repository — and greatfire.org, which distributes software to help Chinese citizens evade censorship restrictions enacted by the so-called “Great Firewall of China,” found themselves on the receiving end of a massive and constantly-changing attack apparently designed to prevent people from being able to access the sites.
Experts have long known that China’s Great Firewall is capable of blocking Web surfers from within the country from accessing online sites that host content which is deemed prohibited by the Chinese government. But according to researchers, this latest censorship innovation targeted Web surfers from outside the country who were requesting various pages associated with Baidu, such that Internet traffic from a small percentage of surfers outside the country was quietly redirected toward Github and greatfire.org.
This attack method, which the researchers have dubbed the “Great Cannon,” works by intercepting non-Chinese traffic to Baidu Web properties, Weaver explained.
Chillingly, the report concludes that Chinese censors could just have easily served malicious code to exploit known Web browser vulnerabilities.
“With a minor tweak in the code, they could have provided exploits to targeted [Internet addresses], so that instead of intercepting all traffic to Baidu, they would serve malware attacks to those visitors,” Weaver said.
Interestingly, this type of attack is not unprecedented. According to documents leaked by National Security Agency whistleblower Edward Snowden, the NSA and British intelligence services used a system dubbed “QUANTUM” to inject content and modify Web results for individual targets that appeared to be coming from a pre-selected range of Internet addresses.
“The Chinese government can credibly say the United States has done similar things in the past,” Weaver said. “They can’t say we’ve done large scale DDoS attacks, but the Chinese government can honestly state that the U.S. has modified traffic in-flight to attack and exploit systems.”
Weaver said the attacks from the Great Cannon don’t succeed when people are browsing Chinese sites with a Web address that begins with “https://”, meaning that regular Internet users can limit their exposure to these attacks by insisting that all Internet communications are routed over “https” versus unencrypted “http://” connections in their browsers. A number of third-party browser plug-ins — such as https-everywhere — can help people accomplish this goal.
“The lesson here is encrypt all the things all the time always,” Weaver said. “If you have to worry about a nation state adversary and if they can see an unencrypted web request that they can tie to your identity, they can use that as a vehicle for attack. This has always been the case, but it’s now practice.”
But Bill Marczak, a research fellow with Citizen Lab, said relying on an always-on encryption strategy is not a foolproof counter to this attack, because plug-ins like https-anywhere will still serve regular unencrypted content when Web sites refuse to or don’t offer the same content over an encrypted connection. What’s more, many Web sites draw content from a variety of sources online, meaning that the Great Cannon attack could succeed merely by drawing on resources provided by online ad networks that serve ads on a variety of Web sites from a dizzying array of sources.
“Some of the scripts being injected in this attack are from online ad networks,” Marczak said. “But certainly this kind of attack suggests a far more aggressive use of https where available.”
For a deep dive into the research referenced in this story, check out this link.
The Linux Foundation (LF) has announced
that it will serve as host of the Let’s Encrypt
project, as well as the Internet Security Research Group (ISRG).
Let’s Encrypt is the free, automated SSL/TLS certificate authority
that was announced in November 2014 by the Electronic Frontier Foundation
(EFF) to provide TLS certificates for every domain on the web. ISRG is
the non-profit organization created to spearhead efforts like Let’s
Encrypt (which, as of now, is ISRG’s only public project). In the LF
announcement, executive director Jim Zemlin notes that “by
hosting this important encryption project in a neutral forum we can
accelerate the work towards a free, automated and easy security
certification process that benefits millions of people around the
While video gaming used to be a strictly offline affair, in the current market many titles require continued access to custom online resources in order to function.
Updates, patches and multi-player support aside, some titles simply cease to function when their developers or publishers decide that the game has outlived its usefulness.
While this is convenient for companies looking to promote the latest titles to their customer bases, those who have invested in software are regularly abandoned along with their now-useless games.
In attempt to remedy this situation the Electronic Frontier Federation (EFF) has teamed up with law student Kendra Albert to seek legal protections from the Copyright Office for those who modify gaming code in order to keep titles playable.
The problem lies in the anti-circumvention provisions of the Digital Millennium Copyright Act (Section 1201) which create legal uncertainty for those digging into code for such purposes. To create clarity, provide protection and allow for the functional preservation of videogame art, the EFF is seeking an exemption to the Act.
“Section 1201 is often used by the entertainment industries not to prevent copyright infringement but to control markets and lock out competition. So it’s not surprising that ESA (the trade association for the largest game producers), along with MPAA and RIAA, have written to the Copyright Office to oppose this exemption,” EFF attorney Mitch Stoltz explains.
“They say that modifying games to connect to a new server (or to avoid contacting a server at all) after publisher support ends—letting people continue to play the games they paid for—will destroy the video game industry. They say it would ‘undermine the fundamental copyright principles on which our copyright laws are based’.”
Indeed, the testimony of ESA Senior Vice-President and General Counsel Christian Genetski before the Subcommittee on Courts, Intellectual Property and the Internet last year (pdf), outlines the software group’s position clearly.
“[W]hile addressing copyright infringement is one important objective of Section 1201, it is not its only objective,” Genetski said.
“[A] prohibition on the hacking of technological protection measures controlling access to protected works (even if the hacking does not result in any copyright infringement) [is] necessary in order to encourage innovation in the online distribution of copyrighted works.”
While the ESA appears to have at least drawn a distinction between piracy and non copyright-infringing activity in its 2014 submission, the EFF says that the software group is now using language that closes the gap somewhat.
Any exception to Section 1201’s blanket ban on circumvention would send a message that “hacking — an activity closely associated with piracy in the minds of the marketplace — is lawful”, the ESA says, adding that the same would “undermine the fundamental copyright principles on which our copyright laws are based.”
It’s fair to say that the EFF remains unimpressed by this interpretation.
“Imagine the havoc that could result if people believed that ‘hacking’ was ever legal! Of course, ‘hacking’ is legal in most circumstances,” Stoltz says.
“Most of the programmers that create games for Sony, Microsoft, EA, Nintendo, and other ESA members undoubtedly learned their craft by tinkering with existing software. If ‘hacking,’ broadly defined, were actually illegal, there likely would have been no video game industry.”
In its submission to the U.S. Copyright Office (pdf), the EFF lists dozens of server shutdowns in 2014 alone, affecting titles such as Age of Empires Online, various Battlefield, Command and Conquer and Crysis titles, several FIFA, Madden and Mario games, plus more than a dozen Pokemon editions.
While these titles have been committed to the graveyard for now, the EFF hopes that an exemption to the DMCA will allow them to enjoy new life. They are supported by T.L. Taylor, Associate Professor of Comparative Media Studies at the Massachusetts Institute of Technology.
“The preservation of computer games includes not only making sure we can see their graphics or hear their sounds, but understand the complexity of their mechanics. Given the market life-cycle of most games, protection is needed to ensure research can continue on these artifacts even after developers have moved onto other ventures,” Taylor writes.
“I believe the exemption proposed here offers a critical path to supporting a range of work that, far from harming any stakeholders, fosters the lively use, development, and scholarship of digital gaming.”
The Federal Bureau of Investigation (FBI) is warning that individuals sympathetic to the Islamic State of Iraq and al-Shams (ISIS) are mass-defacing Websites using known vulnerabilities in WordPress. The FBI also issued an alert advising that criminals are hosting fraudulent government Web sites in a bid to collect personal and financial information from unwitting Web searchers.
According to the FBI, ISIS sympathizers are targeting WordPress Web sites and the communication platforms of news organizations, commercial entities, religious institutions, federal/state/local governments, foreign governments, and a variety of other domestic and international sites. The agency said the attackers are mainly exploiting known flaws in WordPress plug-ins for which security updates are already available.
The public service announcement (PSA) coincides with a less public alert that the FBI released to its InfraGard members, a partnership between the FBI and private industry partners. That alert noted that several extremist hacking groups indicated they would participate in an operation dubbed #OpIsrael, which will target Israeli and Jewish Web sites to coincide with Holocaust Remembrance Day (Apr .15-16).
“The FBI assesses members of at least two extremist hacking groups are currently recruiting participants for the second anniversary of the operation, which started on 7 April 2013, and coincides with Holocaust Remembrance Day,” the InfraGard alert notes. “These groups, typically located in the Middle East and North Africa, routinely conduct pro-extremist, anti-Israeli, and anti-Western cyber operations.”
Experts say there may be no actual relationship between these defacements and Islamist militants. In any case, if you run a Web site powered by WordPress — or any other content management system (CMS) — please take a few moments today to ensure that the CMS itself is up-to-date with the latest patches, and apply all available fixes for any installed plug-ins.
The FBI also issued an unrelated PSA advising people to be wary of fake government Web sites set up to take advantage of search engine optimization techniques that try to get the sites listed prominently in search results when searching for government services online. The FBI explains the scam thusly:
“Victims use a search engine to search for government services such as obtaining an Employer Identification Number (EIN) or replacement social security card. The fraudulent criminal websites are the first to appear in search results, prompting the victims to click on the fraudulent government services website. The victim completes the required fraudulently posted forms for the government service they need. The victim submits the form online, believing they are providing their PII to government agencies such as the Internal Revenue Service, Social Security Administration, or similar agency based on the service they need.”
“Once the forms are completed and submitted, the fraudulent website usually requires a fee to complete the service requested. The fees typically range from $29 to $199 based on the government service requested. Once the fees are paid the victim is notified they need to send their birth certificate, driver’s license, employee badge, or other personal items to a specified address. The victim is then told to wait a few days to several weeks for processing.”
“By the time the victim realizes it is a scam, they may have had extra charges billed to their credit/debit card, had a third-party designee added to their EIN card, and never received the service(s) or documents requested. Additionally, all of their PII data has been compromised by the criminals running the websites and can be used for any number of illicit purposes. The potential harm gets worse for those who send their birth certificate or other government-issued identification to the perpetrator.”
The FBI advises consumers to use search engines or other websites to research the advertised services or person/company you plan to deal with. Search the Internet for any negative feedback or reviews on the government services company, their Web site, their e-mail addresses, telephone numbers, or other searchable identifiers. Fly-by-night scam Web sites often have little or no reputation — i.e., they haven’t been online that long. A simple WHOIS Web site registration record search will often reveal scam domains as just recently having been put online.
SANS Internet Storm Center, InfoCON: green: Guest Diary: Xavier Mertens – Analyzing an MS Word document not detected by AV software, (Tue, Apr 7th)
[Guest Diary: Xavier Mertens] [Analyzing an MS Word document not detected by AV software]
Like everybody, Im receiving a lot of spam everyday but… I like it! All unsolicited received messages are stored in a dedicated folder for two purposes:
- An automatic processing via my tool mime2vt (http://blog.rootshell.be/2014/12/15/automatic-mime-parts-scanning-with-virustotal/)
- A manual review at regular interval
This helps me to find new types of spams or new techniques used by attackers to deliver malicious content in our mailboxes. Today, I received an interesting Word document. Im not sure if it is a very common one but I did a small analysis. The mail was based on a classic fake invoice notification:
From: Ollie Oconnor
Subject: 49933-Your Latest Documents from RS Components 570009054
Alex Stanford – GIAC GWEB GSEC,
Research Operations Manager,
SANS Internet Storm Center
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
However, there are also parts of the world where piracy is frequently used as a means to gather and spread knowledge. In parts of Africa, for example, where legal access to educational books and software is often restricted or unavailable.
Over the years we have seen various illustrations of the educational importance of piracy in developing countries. When the e-book portal Library.nu was shut down, for instance, we were contacted by a United Nations worker in Kenya, who voiced his disappointment.
“I am very concerned about the recent injunction against library.nu. The site was particularly useful for people like me working in Nairobi, a city that has no more than four bookshops with nothing but bestsellers,” the UN worker informed TF at the time.
In an effort to determine how piracy affects literacy and the spread of knowledge, the African Governance and Development Institute conducted an in-depth study comparing piracy and human development data from 11 African countries.
The findings, presented in a paper titled “The Impact of Software Piracy on Inclusive Human Development: Evidence from Africa” show that “software piracy increases literacy”.
“Adoption of tight IPRs regimes may negatively affect human development by diminishing the literacy rate and restricting diffusion of knowledge,” the authors write.
Not all copyright protection measures have a negative effect though, and the researchers found that is negatively linked to the human development index.
“Adherence to international IPRs protection treaties (laws) may not impede per capita economic prosperity and could improve life-expectancy,” the paper reads.
The paper reports mostly correlational data so it’s not unthinkable that countries where human development is higher have less need to pirate, as there are better alternatives.
The reverse effect could also apply to the literacy findings but according to the researchers this is unlikely. Researcher Simplice Asongu informed TF that his previous work showed a causal effect from piracy on scientific publications.
“I tested the impact of piracy on scientific publications and established a positive causality flowing from the former to the latter,” Asongu says.
From that research, it was concluded that African countries with less copyright restrictions on software will substantially boost the spread of knowledge through scientific and technical publications.
The findings reported here are limited to the effect of software piracy, but it’s not hard to see how book piracy may also positively influence literacy and the spread of knowledge.
In sum, the research suggests that piracy does have its positive sides, especially in terms of human development. Still, it seems unlikely that rightsholders will take that into account when lobbying for new policy changes.
Водих лекция на темата инфраструктура на “дни на софтуерната архитектура”. Нещо не се хареса на публиката, но се надявам да е по-смилаема в текстов вид.
Ето и презентацията.
[slide 2 Кой съм аз]
Казвам се Васил Колев и съм дошъл да ви говоря по темата за инфраструктурата.
Аз съм системен администратор, и тук на слайда може да видите някакво описание на нещата, с които съм се занимавал. Като цяло съм си играл и с малки и големи неща, в различни области и съм имал досег с всякакви странни неща, вкл. такива, които ме е срам да си напиша в презентацията (държавни и финансови неща).
Накратко, аз съм човекът, който ще бъде събуден в 4 сутринта, ако нещо не работи. Това ми докарва голямо желание всичко да работи максимално безпроблемно, защото аз доста обичам да спя…
[slide 3 За какво ще ви говоря]
Това е кратък списък на нещата, за които ще ви говоря днес – всичките са неща, които съм срещал в моята практика и с които съм се борил (и още се боря). Повечето от тях са тривиални за решаване, някои не толкова, но като цяло всички продължават да се срещат, въпреки че поне някои от тях са известни от десетилетия.
[slide 4 За какво няма да говоря]
Storage в последните години толкова много се променя, че каквото и да ви говоря за него, може да се окаже съвсем грешно утре.
Също така няма да ви говоря какво е това облак и колко е полезно, велико, важно и т.н.. Ако се интересувате, написал съм отдолу дефиницията…
[slide 5 Какво имам в предвид под “state”]
И ще искам да изясня един-два термина, които ще споменавам от време на време в лекцията си.
Първото е “state”, на български “състояние” – това са всичките данни на една система, реално погледнато всичко без кода и хардуера. Това е важната част на системата за нейните потребители, другите части са интересни на вас и на мен. Този state създава доста проблеми и сигурно щеше да е много по-лесно, ако го нямаше…
[slide 6 Какво е инфраструктура?]
Терминът “инфраструктура” буквално значи “структурата отдолу”, т.е. това, на което се крепите (например подът и столовете в залата са инфраструктура, както и климатизацията и, електричеството и т.н.). Може да мрежите, които ползвате, може да е директно разни услуги (всичките “cloud” неща могат да се кръстят “infrastructure as a service”), но е нещо, което не контролирате пряко и на което разчитате.
То може да е и на сравнително високо ниво, като база данни, web или application сървър, в зависимост от това вие къде се намирате.
Нещата, за които ще говоря важат за по-голямата част от тази инфраструктура, като една прилична част се отнася нивото към сървърите, в/у които работите.
[slide 7 CAP теорема]
И нещо доста важно, т.нар. CAP теорема – която казва, че ако имате дистрибутирана система за съхраняване на state (т.е. например база данни), тя може да има само две от следните три свойства: консистентност (данните да са същите и на двете места), достъпност (винаги да работи) и partition tolerance (да може да работи при разцепване). Има някои подробности по дефиницията на консистентност и доколко съвпада с дефиницията от реалния свят, но за нашите цели теоремата важи в този си вид, и накратко ни казва “трудно е да се направи дистрибутирана база данни”.
[slide 8 ]
И нека покажа две примерни системи. Ето така изглеждат на хартия стандартните архитектури на разни услуги, да речем web-базирани.
Забелязвате ли как съм кръстил схемите “сферичен кон”? Това, защото в реалния свят системите изглеждат ето така:
[slide 9 ]
Това е система, чиято схема рисувах при последния проект, с който се занимавах. Така изглеждат нещата в реалния свят, на нормален не-особено-голям проект, който си е растял органично няколко години – не е нещо, което е направено зле, писано е както трябва от умни хора.
[slide 10 Каква е моята работа]
Та, моята работа е да карам ето такива системи да работят, като им планирам и изграждам така инфраструктурата, че да преживяват всякакви проблеми. Около това съм се нагледал на проблемите, за които ще ви говоря.
[slide 11 ]
Single points of failure са може би основния проблем, който съм виждал в практиката.
[slide 12 ]
На този пример можем да видим системи, които са в общи линии изградени от такива точки – отпадането на който и да е компонент или връзка ще спре работата на системата. Това в общи линии е стандартното нещо, което съм виждал.
Класическият случай на това е всички тези компоненти да са в/у един сървър, в който случай целия му хардуер и софтуер се превръщат в нещо, чието отпадане убива услугата.
[slide 13 Как се създават в софтуера]
Единствената причина да съществуват е, че в тях има local state. Ако нямате потребителски данни в някой компонент, е тривиално да го дублирате и скалирате почти колкото си искате, без да имате сериозни проблеми.
На теория това трябва да е просто – всичките учебници казват да си държите всичките данни в базата. На практика обаче се оказва, че дори сериозни и добри програмисти започват да държат сесии, разни ticket-и и такива неща я локално на диска, я директно в паметта, което от своя страна прави голям проблем съществуването на две копия на компонента.
[slide 14 Как се избягват]
Избягването им не е сложно. След като се запознаете с CAP теоремата и преживеете ужасните и последици, намирате подходящ компромис за съхраняването на state, изнасяте го там и дублирате останалото.
В наше време това е все по-лесно и по-лесно. В последния проект имаше такъв случай с потребителски сесии и още някакви неща, и се оказа че с 20на реда код проблемът се премахва, като всичките тези данни отидат в базата.
(нищо общо с времената, в които се налагаше да препишем половината код при подобна промяна)
[slide 15 Дистрибутиран/подсигурен state (1/2)]
Дистрибутираният state не е тривиална задача. Това, което винаги работи (и което се използва в крайни случаи, е синхронно репликирано копие на базата (или каквото-там-ни-съхранява данните), което да поеме работата на главното, ако то отпадне. Това върши работа за почти всичко, но не се поддава на скалиране и е подходящо основно ако нямате голямо натоварване на системата. Това е setup, който се среща прилично често при финансови институции, понеже е стар колкото света, много добре разбран и лесно може да се приложни за всичко.
[slide 16 Дистрибутиран/подсигурен state (2/2)]
Вариантът да си смените базата или да я разширите до нещо, което се скалира/дистрибутира е един от най-хубавите, за съжаление и от най-трудните. CAP теоремата ни създава доста ограничения, физиката ни създава други, и в крайна сметка се налага да правим компромиси или да променяме цялостната архитектура. В наши дни има доста eventual consistency бази, както и всякакви NoSQL решения, голяма част от които са си жива подигравка с идеята за бази данни, но някои от тях са сравнително използваеми.
И да, всички искаме една безкрайна SQL ACID база данни, но такава няма – дори това, което ни предлагат за огромни пари има някакви странни ограничения, в които се удряме докато ги ползваме.
Няма да продължавам с въпроса, понеже вероятно може да се запълни едно-семестриален курс с него, вероятно и повече.
[slide 17 Всичко останало]
Та, както казах всичко останало се решава лесно с дублиране и load balancing.
[slide 18 ]
На схемата може да се види един не-толкова-сферичен кон, просто дублирана система, при която няма какво да отпадне, че да ни направи генерален проблем (може би с изключение на връзката към internet :) ).
[slide 19 TL;DR]
Та, няколко кратки извода – ползвайте подходящото място да си държите данните, не се ограничавайте какво да ползвате, и не искайте невъзможното.
[slide 20 ]
Сравнително близък до предния проблем е хоризонталния scaling, т.е. възможността да увеличите колко натоварване може да издържи системата чрез добавяне на още хардуер към нея (вместо с upgrade-ване на текущия).
[slide 21 ]
Ето една тъжна картинка, която показва как производителността на един core на процесор не се увеличава толкова много, колкото едно време, т.е. клоним все повече и повече към повече ядра на процесор, отколкото към по-бързи ядра. Въпреки всичкия research и мъки, физиката и някои математически проблеми ни спират да постигнем по-голяма производителност.
[slide 22 Накъде вървим]
Това ще ни доведе до големи клъстери от памет, без споделена памет, или с бавен достъп до споделената памет. Все повече и повече и инфраструктурата, която се предоставя е такава – забележете как това, което amazon ви предлага са много отделни не-особено-мощни виртуални машини, които обаче може да вдигате в огромни количества.
Може да очаквате да има още по-голяма нужда от паралелизация на нещата, които правите, като не-паралелните ви задачи да трябва да са малки и кратки, за да не се създават ненужни закъснения и забавяния.
Като пример мога да дам, че в момента един от най-бързите архитектурни модели за правене на сървъри, които process-ват много заявки е на принципа на co-routines, познати в windows-кия свят като “fibers”, а в erlang като “actors”. Идеята е, че вместо да имате цели thread-ове, които да изпълняват всеки по дадена задача, имате гигантско количество малки задачки, които се schedule-ват и изпълняват когато им е възможно. В книгата “The performance of open-source applications” има два такива примера (за съжаление на erlang и на haskell), но ако някой се интересува, мога да разкажа за такова решение с човешки езици за програмиране, която трябва да се появи като open-source някой ден.
[slide 23 Какво следва]
Това ни води до няколко важни неща, идващи от паралелното програмиране.
Трябва да избягваме бавните непрекъсваеми процеси и да оптимизираме за латентност. Дълги години слушах “защо да оптимизирам, те компютрите стават по-бързи”, но това вече не е факт, този процес се забавя и вече не можем да разчитаме на закона на Мур да ни спаси.
Трябва да избягваме зависимостите, доколкото можем, както и създаването на bottlenecks – единични компоненти, през които трябва да мине всичко (и които освен това ще са single points of failure).
[slide 24 ]
Нещо, което хората не разбират, са мрежите. Не е като да са много нови, но липсва яснота доколко зависим от тях и къде могат да ни спънат.
[slide 25 Всичко е мрежа]
Например вече всичко е мрежа. От вътрешната архитектура на процесорите, през дънните плати, SATA/SAS – всичките тези технологии представляват някакви мрежи в звездоподобна технология, със switch-ове и router-в тях, по някакви собствени протоколи. Да не говорим, че вече почти нищо не ни е локално (например базата данни ще ви е на същата машина само ако сте много малка услуга).
Като комбинираме това с навлизането на всякаквите облачни услуги, вече остават малко неща, за които да не зависим от мрежата.
[slide 26 List of network fallacies]
И ще ви говоря по този списък, който е от около 1994та, но по някаква причина не се преподава в училище.
[slide 28 Internet и мрежите не са reliable]
Да почнем от там, че internet и мрежите като цяло не са reliable. Нямате гаранция за нищо, всичко е best-effort и за да постигнете някаквите си нужни гаранции е необходимо да вкарате логика при вас, която да се справя с такива проблеми.
Ужасно важно е да не разчитате много на мрежата. Виждал съм (при едни хора, които са в top 100 на посещаемост в internet) да се синхронизират бази данни през internet, което водеше до всякакви странни неконсистентности и голямо количество логика, което да се бори с тях.
По същия начин ако имате каквато и да е синхронна операция през internet (например да отбележите някъде брояч или да проверите нещо, преди да отговорите на потребителя), това ще ви създава от време на време сериозни проблеми и ще е трудно да се хване откъде идват.
[slide 29 Да забравим латентността]
Дори да нямаме проблеми със загуби по мрежата, самата мрежа има някаква латентност в себе си, която освен че не е постоянна, доста често е по-голяма, отколкото сме свикнали при директната комуникация. Тази латентност се натрупва, и често при малка промяна на ситуацията може да ни създаде големи проблеми.
Един прост пример, който мога да дам е, че по принцип SSL протокола прави от 3 до 6 round-trip-а, т.е. отивания на пакети от единия до другия край, за да осъществи връзка. По принцип ако си говорите със сървър в същата държава, едно RTT е под 20ms, и това няма да се усети лесно като проблем, но ако говорите от Европа до Щатите и едно rtt е 150ms, изведнъж тази разлика става наистина осезаема – потребителите започват да усещат забавянето и да не искат да ви ползват сайта. Това може да се реши в софтуера с настройка на самия SSL протокол, но малко хора се занимават да му обръщат внимание.
[slide 30 Твърде много данни по твърде малка тръба]
Трудно ще намерите мрежа, която да достига скоростта на копирането в паметта. Лоша идея е да се опитвате да прекарвате много данни от едно място, или като цяло да прекарвате ненужно количество данни през някакво тясно място от мрежата.
Има неща от типа на remote dma – т.е. да накарате две системи да си предават данни, вместо да ги копирате през себе си. Може да погледнете bittorrent протокола и какво умее – в момента това е най-бързия начин за прехвърляне на информация до много места в internet.
[slide 31 Голи данни по пробита мрежа]
За всички, които не знаят кой е Edward Snowden, моля да отворят wikipedia и да погледат. За останалите – вече се видя колко тривиално се подслушват данни, т.е. и обществото го знае, та няма нужда да обясняваме на всички колко страшни са мрежите и можем да заделим нужния ресурс да криптираме.
На теория трябва да се криптират само важните данни, но понеже няма начин да се определи кои данни са важни, е далеч по-лесно и по правилно да се криптира всичко.
Също така, криптирането изисква познаване на схемите и криптографията като цяло, и препоръчвам всички да се запознаете с подробностите – нещата не са прости, но могат да ви докарат неочаквани екстри, като например силна проверка за валидността на данните.
[slide 32 Мрежата ще отговаря на всичките ви изисквания]
Мрежата е голямо животно. Не цялата се контролира от едно място и рядко е достатъчно хомогенна, за да предполагате какво може. Препоръчвам ви да знаете какво се случва с нея и как изглежда, преди да проектирате нещо, което трябва да работи в нея.
[slide 33 … и т.н.]
Мрежите са нещо прекрасно, те са бъдещето, пътя, истината и живота :) Могат много да ви помогнат, могат да ви съсипят и е важно да ги познавате в подробности.
[slide 34 ]
И ще поговоря малко по темата дизайн на системи, понеже има голямо отношение към инфраструктурата и комуникацията с нея. Ще започна с нещо просто, разликата м/у store&forward архитектури и pass-through.
[slide 35 Дефиниция]
Това е доста често срещан проблем – дали да съхраним данните, или да ги обработваме докато идват. Това зависи от данните и протоколите, но има и сериозно отношение към reliability-то на системата.
[slide 36 Примерна система]
Примерът, който ще дам е с една проста услуга, която получава писма и ги обработва. Можем да я реализираме по два начина – да получим писмото, да го съхраним и да го предаем за обработка, или да започнем да го предаваме, докато още го получаваме.
[slide 37 Предимства и недостатъци]
Тук и двата начина си имат предимства и недостатъци. Pass-through има по-ниска латентност, което понякога може да се окаже наистина важно, но от друга страна може да се претовари по-лесно и е по-чупливо от store&forward. Pass-through и са по-сложни за дистрибутиране.
[slide 38 Извод]
Pass-through като цяло е доста по-ефективно решение, ако ви е нужна скорост, но препоръчвам store&forward за всичко, което не е критично – може да издържи доста повече на проблеми и по-лесно се възстановява след проблеми.
[slide 40 ]
Ако гледаме дизайна на системи, в общи линии имаме два варианта – линейни и комплексни. В линейните всичко върви в едната или другата посока и няма сложни интеракции, в комплексната има.
[slide 41 (почти) Всичките ни системи са комплексни]
Ние почти нямаме линейни системи в практиката. Всичко, което правим има странни интеракции в него и е огромно като размер. Дал съм пример как Airbus A380 без да броим софтуера има 4 милиона компонента, а само linux kernel-а е над 12 милиона реда код, или ако го компилираме, в себе си има над милион условни прехода.
Ако хванем всичко, което ползваме, ще се окаже гигантско. За сравнение с нещо, което повече разбираме – един двигател с вътрешно горене е под 1000 компонента (даже като броим всяко винтче и гайка).
[slide 42 Тъжни факти]
Ние на практика нямаме толкова голяма система, която наистина да разбираме, която да може да бъде разбрана от един човек – а от друга страна очакваме, че всеки един програмист може да се справи със съществуващите ни системи. Като комбинираме това, че средно имаме по 10-15 грешки на 1000 реда код в изтествани и release-нати продукти е учудващо, че нещо изобщо работи.
В механичния свят е по-лесно да направим система, с чиито failure mode-ове да сме наясно при отпадане на един или два компонента – например сме сравнително наясно с двигателите на колите, но реално погледнато всичко по-сложно от това ни е сложно.
[slide 43 Може да се мине със следния цитат]
Винаги обичам да давам този цитат. За който не е запознат, Jim Gray е човекът, измислил голяма част от теорията за backend-ите на базите данни, които ползваме в наши дни (например цялата ACID идея).
[slide 44 Изроди]
Изводите мисля, че са ясни, нека да правим прости неща, доколкото можем, и да се стремим към простота. Неколкократно съм виждал система да бъде реализирана добре в 10к реда код, и лошо в 100к, и си мисля, че си заслужава усилието да правим минималистично нещата.
[slide 45 ]
И ще приключа с нещо наистина страшно.
[slide 46 Какво е bitrot]
Bitrot е повреждането на данните, без вие да знаете. Очаквате, че нещо се е записало правилно, че е ок, след което го прочитате и доста пъти дори без да разберете в него има някакви повреди.
[slide 47 Статистика от wikipedia]
От wikipedia може да видите ето тези прости статистики, които са доста неприятни, но не са достатъчно близки до вас, за това ето няколко, с които директно съм се сблъскал.
[slide 48 Случки от реалния свят]
В една система с 395 милиона файла имаха около 300 грешки в рамките на месец, и в последната седмица – 3 (това беше проверка по sha1 checksum-а на файловете). Това са свестни, малко по-стари сървъри, с raid5 масиви, а софтуерът е съвсем малко и много внимателно изтестван и прегледан – и все пак се появяват такива проблеми. Скоро 1PB няма да е огромно число, а нещо нормално и очаквано, и можем да очакваме при всички ни да се видят такива проблеми.
И нещо още по-неприятно – едни приятели имат софтуер, на който правят сериозни regression test-ове. В техния lab се появили някакви crash-ове, които се появявали по странно време без никаква логика, и след известно време открили, че това се случва само на сървърите без ECC памет. Машините имали 16-32 GB памет и не са били купувани на битака :)
[slide 49 row-hammer exploit]
И, нещо съвсем скорошно, т.нар. row-hammer exploit, че ако пишете много някъде по паметта, можете да flip-нете бит на друго място. С това даже бяха успели да напишат и exploit, с който да може през browser-а и sandbox-а му да се излезе и да се получи root достъп.
[slide 50 Инструкции за цивилното население]
Това е ужасно.
Няма хубаво решение на тия проблеми – реално няма много research как да работим със счупен хардуер – но трябва да знаем, че ще ни се случи. Решенията, които имам са съвсем малко (освен първото) – максимално прости системи, failfast – т.е. да се спира при първия възможен шанс, ако се открие нещо такова, и run-time проверки за коректност (познати ви като assert-и), за да може да се ограничат последиците от проблема.
[slide 51 ]
И няколко книги, които могат да са ви полезни:
* “The Architecture of Open-source Applications”, vol. 1 и 2
* “The Performance of Open-source Applications”
* “Beautiful Data”
* “Normal Accidents: Living with High-Risk Technologies”, Charles Perrow
* “Cryptography Engineering”, Schneier, Ferguson & Konho
The Rust team at Mozilla Research has announced the first beta release of Rust 1.0. The release notes detail a number of important changes, but the announcement adds some additional noteworthy items. “The Beta release also marks a turning point in our approach to stability. During the alpha cycle, the use of unstable APIs and language features was permitted, but triggered a warning. As of the Beta release, the use of unstable APIs will become an error (unless you are using Nightly builds or building from source).” A new continuous-integration infrastructure has also been deployed. The final release is currently expected around May 15.
In many countries there are exceptions to copyright law that allow those in education to use copyrighted material to further their studies.
Those exceptions often have limits but copying for research, comment and reporting purposes are generally allowed while teachers are able to make multiple copies of content to hand out to their students.
Following the tabling of a new intellectual property law in Spain, last December the Department of Education sent out a circular reminding schools that the showing of audiovisual content outside strict “fair use” parameters is completely banned.
While airing short clips should be ok, the government had become concerned that schools stepping over the mark could be forced to obtain prior authorization to show content or might even find themselves being sued. That resulted in the decision-making body in the autonomous region of Galicia striking a private licensing deal with rightsholders from the movie industry.
The letter revealed that MPLC was willing to license each student for the price of 1.25 euros per year. While that doesn’t sound much in isolation, there are 260,000 students in the region making a grand total of 325,000 euros ($350,000) to be sent to MPLC’s movie and TV show company members.
The CIG-Ensino union has reacted furiously to the news and is now calling for local authorities to prohibit the collection of any monies and ensure that audiovisual resources for use as teaching and learning aids remain free.
“[Schools and teachers] should not to pay any tax for doing their job and should be able to continue using all kinds of tools that are needed to do their jobs as effectively as possible,” the union said.
“It is incomprehensible to try to limit the task of educating exclusively to the use of the textbooks and reducing the use of resources such as film, music, documentaries in classrooms.”
MPLC has not yet commented on the news.
|A typical hacker, according to @Viss|
President Obama has upped his war on hackers by declaring a “state of emergency“. This triggers several laws that grant him expanded powers, such as seizing the assets of those suspected of hacking, or taking control of the Internet.
One one hand, this seems reasonable. Hackers from China and Russia are indeed a threat, causing billions in economic damage every year, by stealing money and intellectual property. This declaration specifically targets these issues. Presumably, in the next few weeks, we’ll see announcements from the Treasure Department seizing assets from Chinese companies known to have stolen intellectual property via hacking.
But on the other hand, it’s problematic. Declarations of emergency tend to be permanent. We already operate under 30 declarations of emergencies dating back to the Korean war. Once government grabs new powers, it tends not to give them back. Also, this really isn’t an “emergency”, the hacking it addresses goes back a decade. It’s obvious corruption of the “emergency” provisions in the law for the President to bypass congress and rule by decree.
Moreover, while tailored specifically to the threats of foreign hackers, it ultimately affects everyone everywhere. It allows the government to bypass due process and seize the assets of anybody suspected of hacking. The federal government already widely abuses “asset forfeiture” laws, seizing a billion dollars annually. This executive order expands such activities (although “freezing” isn’t quite the same as “forfeiture”).
Of particular concern are “security researchers”. The only way to secure systems is to attack them. Securing systems means pointing out flaws, which inevitably embarrasses the powerful, who then lobby government for assistance in dealing with these pesky “hackers”.
The White House knows this is a potential problem, and clarifies that it doesn’t intend to use this Executive Order to go after security researchers. But this is bogus. Whether somebody is a “good guy” or a “bad guy” is merely a matter of perspective. For example, I regularly scan the entire Internet. The security research community broadly agrees this is a good thing, but the powerful disagree. I have to exclude the DoD from my scans, because they make non-specific threats toward me in order to get me to stop. This Executive Order makes those threats real — giving the government the ability to declare my scans “malicious” and to seize all my assets. It’s the Treasury Department who makes these decisions — from their eyes, “security research” is indistinguishable from witchcraft, so all us researchers are malicious.
This last week, we saw a DDoS attack by China against a key Internet infrastructure company known as “GitHub”. The evidence clearly points to the Chinese government as the culprit — yet the President has remained silent on the issue. In contrast, the President readily spoke out against North Korea based on flimsy evidence. These new powers granted by the Executive Order do nothing to stop such an attack. With proposed laws, such as CISA surveillance expansion law, or the extensions to the CFAA, we see that the government is eager to obtain new powers, but reluctant to actually use the powers it already has to defend against hackers.
The reason the government is hesitant is that China is a thorny problem. North Korea is an insignificant country, so we bully them whenever it’s convenient. In contrast, China’s economy rivals our own. Moreover, trade intertwines our economies. Logical next steps to address hacking involve economic sanctions that will hurt both countries. What the government will do to address Chinese hacking then becomes a political question. No matter how many powers we give government, no matter how much we sacrifice privacy rights, stopping foreign hackers becomes a political question of foreign policy.
The conclusion is this: from the point of view of government, this Executive Order (and the follow-on actions by the Treasury Department) are a reasonable response to recent hacking. But the reality is that it’s a power grab by government, granting them new powers to bypass our rights, that they are unlikely to ever give up. It’s unlikely to solve the problem of foreign hacking, but will do much to expand the cyber police state.
Pew Research has a new survey on Americans’ privacy habits in a post-Snowden world.
The 87% of those who had heard at least something about the programs were asked follow-up questions about their own behaviors and privacy strategies:
34% of those who are aware of the surveillance programs (30% of all adults) have taken at least one step to hide or shield their information from the government. For instance, 17% changed their privacy settings on social media; 15% use social media less often; 15% have avoided certain apps and 13% have uninstalled apps; 14% say they speak more in person instead of communicating online or on the phone; and 13% have avoided using certain terms in online communications.
25% of those who are aware of the surveillance programs (22% of all adults) say they have changed the patterns of their own use of various technological platforms “a great deal” or “somewhat” since the Snowden revelations. For instance, 18% say they have changed the way they use email “a great deal” or “somewhat”; 17% have changed the way they use search engines; 15% say they have changed the way they use social media sites such as Twitter and Facebook; and 15% have changed the way they use their cell phones.
Also interesting are the people who have not changed their behavior because they’re afraid that it would lead to more surveillance. From pages 22-23 of the report:
Still, others said they avoid taking more advanced privacy measures because they believe that taking such measures could make them appear suspicious:
“There’s no point in inviting scrutiny if it’s not necessary.”
“I didn’t significantly change anything. It’s more like trying to avoid anything questionable, so as not to be scrutinized unnecessarily.
“[I] don’t want them misunderstanding something and investigating me.”
There’s also data about how Americans feel about government surveillance:
This survey asked the 87% of respondents who had heard about the surveillance programs: “As you have watched the developments in news stories about government monitoring programs over recent months, would you say that you have become more confident or less confident that the programs are serving the public interest?” Some 61% of them say they have become less confident the surveillance efforts are serving the public interest after they have watched news and other developments in recent months and 37% say they have become more confident the programs serve the public interest. Republicans and those leaning Republican are more likely than Democrats and those leaning Democratic to say they are losing confidence (70% vs. 55%).
Moreover, there is a striking divide among citizens over whether the courts are doing a good job balancing the needs of law enforcement and intelligence agencies with citizens’ right to privacy: 48% say courts and judges are balancing those interests, while 49% say they are not.
At the same time, the public generally believes it is acceptable for the government to monitor many others, including foreign citizens, foreign leaders, and American leaders:
- 82% say it is acceptable to monitor communications of suspected terrorists
- 60% believe it is acceptable to monitor the communications of American leaders.
- 60% think it is okay to monitor the communications of foreign leaders
- 54% say it is acceptable to monitor communications from foreign citizens
Yet, 57% say it is unacceptable for the government to monitor the communications of U.S. citizens. At the same time, majorities support monitoring of those particular individuals who use words like “explosives” and “automatic weapons” in their search engine queries (65% say that) and those who visit anti-American websites (67% say that).
Overall, 52% describe themselves as “very concerned” or “somewhat concerned” about government surveillance of Americans’ data and electronic communications, compared with 46% who describe themselves as “not very concerned” or “not at all concerned” about the surveillance.
It’s worth reading these results in detail. Overall, these numbers are consistent with a worldwide survey from December. The press is spinning this as “Most Americans’ behavior unchanged after Snowden revelations, study finds,” but I see something very different. I see a sizable percentage of Americans not only concerned about government surveillance, but actively doing something about it. “Third of Americans shield data from government.” Edward Snowden’s goal was to start a national dialog about government surveillance, and these surveys show that he has succeeded in doing exactly that.
- 1 -
Poland is an ancient country whose history is deeply intertwined with that of the western civilization. In its glory days, the Polish-Lithuanian Commonwealth sprawled across vast expanses of land in central Europe, from Black Sea to Baltic Sea. But over the past two centuries, it suffered a series of military defeats and political partitions at the hands of its closest neighbors: Russia, Austria, Prussia, and – later – Germany.
After more than a hundred years of foreign rule, Poland re-emerged as an independent state in 1918, only to face the armies of Nazi Germany at the onset of World War II. With Poland’s European allies reneging on their earlier military guarantees, the fierce fighting left the country in ruins. Some six million people have died within its borders – more than ten times the death toll in France or in the UK. Warsaw was reduced to a sea of rubble, with perhaps one in ten buildings still standing by the end of the war.
With the collapse of the Third Reich, the attendees of the Yalta Conference decided the new order of the post-war Europe. At Stalin’s behest, Poland and its neighboring countries were placed under Soviet political and military control, forming what has become known as the Eastern Bloc.
Over the next several decades, the Soviet satellite states experienced widespread repression and economic decline. But weakened by the expense of the Cold War, the communist chokehold on the region eventually began to wane. In Poland, the introduction of martial law in 1981 could not put an end to sweeping labor unrest. Narrowly dodging the specter of Soviet intervention, the country regained its independence in 1989 and elected its first democratic government; many other Eastern Bloc countries soon followed suit.
Ever since then, Poland has enjoyed a period of unprecedented growth and has emerged as one of the more robust capitalist democracies in the region. In just two decades, it shed many of its backwardly, state-run heavy industries and adopted a modern, service-oriented economy. But the effects of the devastating war and the lost decades under communist rule still linger on – whether you look at the country’s infrastructure, at its socrealist cityscapes, at its political traditions, or at the depressingly low median wage.
When thinking about the American involvement in the Cold War, people around the world may recall Vietnam, Bay of Pigs, or the proxy wars fought in the Middle East. But in Poland and many of its neighboring states, the picture you remember the most is the fall of the Berlin Wall.
- 2 -
I was born in Warsaw in the winter of 1981, at the onset of martial law, with armored vehicles rolling onto Polish streets. My mother, like many of her generation, moved to the capital in the sixties as a part of an effort to rebuild and repopulate the war-torn city. My grandma would tell eerie stories of Germans and Soviets marching through their home village somewhere in the west. I liked listening to the stories; almost every family in Poland had some to tell.
I did not get to know my father. I knew his name; he was a noted cinematographer who worked on big-ticket productions back in the day. He left my mother when I was very young and never showed interest in staying in touch. He had a wife and other children, so it might have been that.
Compared to him, mom hasn’t done well for herself. We ended up in social housing in one of the worst parts of the city, on the right bank of the Vistula river. My early memories from school are that of classmates sniffing glue from crumpled grocery bags. I remember my family waiting in lines for rationed toilet paper and meat. As a kid, you don’t think about it much.
The fall of communism came suddenly. I have a memory of grandma listening to broadcasts from Radio Free Europe, but I did not understand what they were all about. I remember my family cheering one afternoon, transfixed to a black-and-white TV screen. I recall my Russian language class morphing into English; I had my first taste of bananas and grapefruits. There is the image of the monument of Feliks Dzierżyński coming down. I remember being able to go to a better school on the other side of Warsaw – and getting mugged many times on the way.
The transformation brought great wealth to some, but many others have struggled to find their place in the fledgling and sometimes ruthless capitalist economy. Well-educated and well read, my mom ended up in the latter pack, at times barely making ends meet. I think she was in part a victim of circumstance, and in part a slave to way of thinking that did not permit the possibility of taking chances or pursuing happiness.
- 3 -
Mother always frowned upon popular culture, seeing it as unworthy of an educated mind. For a time, she insisted that I only listen to classical music. She angrily shunned video games, comic books, and cartoons. I think she perceived technology as trivia; the only field of science she held in high regard was abstract mathematics, perhaps for its detachment from the mundane world. She hoped that I would learn Latin, a language she could read and write; that I would practice drawing and painting; or that I would read more of the classics of modernist literature.
Of course, I did almost none of that. I hid my grunge rock tapes between Tchaikovsky, listened to the radio under the sheets, and watched the reruns of The A-Team while waiting for her to come back from work. I liked electronics and chemistry a lot more than math. And when I laid my hands on my first computer – an 8-bit relic of British engineering from 1982 – I soon knew that these machines, in their incredible complexity and flexibility, were what I wanted to spend my time on.
I suspected I could be a competent programmer, but never had enough faith in my skill. Yet, in learning about computers, I realized that I had a knack for understanding complex systems and poking holes in how they work. With a couple of friends, we joined the nascent information security community in Europe, comparing notes on mailing lists. Before long, we were taking on serious consulting projects for banks and the government – usually on weekends and after school, but sometimes skipping a class or two. Well, sometimes more than that.
All of the sudden, I was facing an odd choice. I could stop, stay in school and try to get a degree – going back every night to a cramped apartment, my mom sleeping on a folding bed in the kitchen, my personal space limited to a bare futon and a tiny desk. Or, I could seize the moment and try to make it on my own, without hoping that one day, my family would be able to give me a head start.
I moved out, dropped out of school, and took on a full-time job. It paid somewhere around $12,000 a year – a pittance anywhere west of the border, but a solid wage in Poland even today. Not much later, I was making two times as much, about the upper end of what one could hope for in this line of work. I promised myself to keep taking courses after hours, but I wasn’t good at sticking to the plan. I moved in with my girlfriend, and at the age of 19, I felt for the first time that things were going to be all right.
- 4 -
Growing up in Europe, you get used to the barrage of low-brow swipes taken at the United States. Your local news will never pass up the opportunity to snicker about the advances of creationism somewhere in Kentucky. You can stay tuned for a panel of experts telling you about the vastly inferior schools, the medieval justice system, and the striking social inequality on the other side of the pond. But deep down inside, no matter how smug the critics are, or how seemingly convincing their arguments, the American culture still draws you in.
My moment of truth came in the summer of 2000. A company from Boston asked me if I’d like to talk about a position on their research team; I looked at the five-digit figure and could not believe my luck. Moving to the US was an unreasonable risk for a kid who could barely speak English and had no safety net to fall back to. But that did not matter: I knew I had no prospects of financial independence in Poland – and besides, I simply needed to experience the New World through my own eyes.
Of course, even with a job offer in hand, getting into the United States is not an easy task. An engineering degree and a willing employer opens up a straightforward path; it is simple enough that some companies would abuse the process to source cheap labor for menial, low-level jobs. With a visa tied to the petitioning company, such captive employees could not seek better wages or more rewarding work.
But without a degree, the options shrink drastically. For me, the only route would be a seldom-granted visa reserved for extraordinary skill – meant for the recipients of the Nobel Prize and other folks who truly stand out in their field of expertise. The attorneys looked over my publication record, citations, and the supporting letters from other well-known people in the field. Especially given my age, they thought we had a good shot. A few stressful months later, it turned out that they were right.
On the week of my twentieth birthday, I packed two suitcases and boarded a plane to Boston. My girlfriend joined me, miraculously securing a scholarship at a local university to continue her physics degree; her father helped her with some of the costs. We had no idea what we were doing; we had perhaps few hundred bucks on us, enough to get us through the first couple of days. Four thousand miles away from our place of birth, we were starting a brand new life.
- 5 -
The cultural shock gets you, but not in the sense you imagine. You expect big contrasts, a single eye-opening day to remember for the rest of your life. But driving down a highway in the middle of a New England winter, I couldn’t believe how ordinary the world looked: just trees, boxy buildings, and pavements blanketed with dirty snow.
Instead of a moment of awe, you drown in a sea of small, inconsequential things, draining your energy and making you feel helpless and lost. It’s how you turn on the shower; it’s where you can find a grocery store; it’s what they meant by that incessant “paper or plastic” question at the checkout line. It’s how you get a mailbox key, how you make international calls, it’s how you pay your bills with a check. It’s the rules at the roundabout, it’s your social security number, it’s picking the right toll lane, it’s getting your laundry done. It’s setting up a dial-up account and finding the food you like in the sea of unfamiliar brands. It’s doing all this without Google Maps or a Facebook group to connect with other expats nearby.
The other thing you don’t expect is losing touch with your old friends; you can call or e-mail them every day, but your social frames of reference begin to drift apart, leaving less and less to talk about. The acquaintances you make in the office will probably never replace the folks you grew up with. We managed, but we weren’t prepared for that.
- 6 -
In the summer, we had friends from Poland staying over for a couple of weeks. By the end of their trip, they asked to visit New York City one more time; we liked the Big Apple, so we took them on a familiar ride down I-90. One of them went to see the top of World Trade Center; the rest of us just walked around, grabbing something to eat before we all headed back. A few days later, we were all standing in front of a TV, watching September 11 unfold in real time.
We felt horror and outrage. But when we roamed the unsettlingly quiet streets of Boston, greeted by flags and cardboard signs urging American drivers to honk, we understood that we were strangers a long way from home – and that our future in this country hanged in the balance more than we would have thought.
Permanent residency is a status that gives a foreigner the right to live in the US and do almost anything they please – change jobs, start a business, or live off one’s savings all the same. For many immigrants, the pursuit of this privilege can take a decade or more; for some others, it stays forever out of reach, forcing them to abandon the country in a matter of days as their visas expire or companies fold. With my O-1 visa, I always counted myself among the lucky ones. Sure, it tied me to an employer, but I figured that sorting it out wouldn’t be a big deal.
That proved to be a mistake. In the wake of 9/11, an agency known as Immigration and Naturalization Services was being dismantled and replaced by a division within the Department of Homeland Security. My own seemingly straightforward immigration petition ended up somewhere in the bureaucratic vacuum that formed in between the two administrative bodies. I waited patiently, watching the deepening market slump, and seeing my employer’s prospects get dimmer and dimmer every month. I was ready for the inevitable, with other offers in hand, prepared to make my move, perhaps the very first moment I could. But the paperwork just would not come through. With the Boston office finally shutting down, we packed our bags and booked flights. We faced the painful admission that for three years, we chased nothing but a pipe dream. The only thing we had to show for it were two adopted cats, now sitting frightened somewhere in the cargo hold.
The now-worthless approval came through two months later; the lawyers, cheerful as ever, were happy to send me a scan. The hollowed-out remnants of my former employer were eventually bought by Symantec – the very place from where I had my backup offer in hand.
- 7 -
In a way, Europe’s obsession with America’s flaws made it easier to come home without ever explaining how the adventure really played out. When asked, I could just wing it: a mention of the death penalty or permissive gun laws would always get you a knowing nod, allowing the conversation to move on.
Playing to other people’s preconceptions takes little effort; lying to yourself calls for more skill. It doesn’t help that when you come back after three years away from home, you notice all the small things you simply used to tune out. The dilapidated road from the airport; the drab buildings on the other side of the river; the uneven pavements littered with dog poop; the dirty walls at my mother’s place, with barely any space to turn. You can live with it, of course – but it’s a reminder that you settled for less, and it’s a sensation that follows you every step of the way.
But more than the sights, I couldn’t forgive myself something else: that I was coming back home with just loose change in my pocket. There are some things that a failed communist state won’t teach you, and personal finance is one of them; I always looked at money just as a reward for work, something you get to spend to brighten your day. The indulgences were never extravagant: perhaps I would take the cab more often, or have take-out every day. But no matter how much I made, I kept living paycheck-to-paycheck – the only way I knew, the way our family always did.
- 8 -
With a three-year stint in the US on your resume, you don’t have a hard time finding a job in Poland. You face the music in a different way. I ended up with a salary around a fourth of what I used to make in Massachusetts, but I simply decided not to think about it much. I wanted to settle down, work on interesting projects, marry my girlfriend, have a child. I started doing consulting work whenever I could, setting almost all the proceeds aside.
After four years with T-Mobile in Poland, I had enough saved to get us through a year or so – and in a way, it changed the way I looked at my work. Being able to take on ambitious challenges and learn new things started to matter more than jumping ships for a modest salary bump. Burned by the folly of pursuing riches in a foreign land, I put a premium on boring professional growth.
Comically, all this introspection made me realize that from where I stood, I had almost nowhere left to go. Sure, Poland had telcos, refineries, banks – but they all consumed the technologies developed elsewhere, shipped here in a shrink-wrapped box; as far as their IT went, you could hardly tell the companies apart. To be a part of the cutting edge, you had to pack your bags, book a flight, and take a jump into the unknown. I sure as heck wasn’t ready for that again.
And then, out of the blue, Google swooped in with an offer to work for them from the comfort of my home, dialing in for a videoconference every now and then. The starting pay was about the same, but I had no second thoughts. I didn’t say it out loud, but deep down inside, I already knew what needed to happen next.
- 9 -
We moved back to the US in 2009, two years after taking the job, already on the hook for a good chunk of Google’s product security and with the comfort of knowing where we stood. In a sense, my motive was petty: you could call it a desire to vindicate a failed adolescent dream. But in many other ways, I have grown fond of the country that shunned us once before; and I wanted our children to grow up without ever having to face the tough choices and the uncertain prospects I had to deal with in my earlier years.
This time, we knew exactly what to do: a quick stop at a grocery store on a way from the airport, followed by e-mail to our immigration folks to get the green card paperwork out the door. A bit more than half a decade later, we were standing in a theater in Campbell, reciting the Oath of Allegiance and clinging on to our new certificates of US citizenship.
The ceremony closed a long and interesting chapter in my life. But more importantly, standing in that hall with people from all over the globe made me realize that my story is not extraordinary; many of them had lived through experiences far more harrowing and captivating than mine. If anything, my tale is hard to tell apart from that of countless other immigrants from the former Eastern Bloc. By some estimates, in the US alone, the Polish diaspora is about 9 million strong.
I know that the Poland of today is not the Poland I grew up in. It’s not not even the Poland I came back to in 2003; the gap to Western Europe is shrinking every single year. But I am proud to now live in a country that welcomes more immigrants than any other place on Earth – and at the end of their journey, makes them feel at home. It also makes me realize how small and misguided must be the conversations we are having about immigration – not just here, but all over the developed world.
Just so you know, x86 machine-code is now a “high-level” language. What instructions say, and what they do, are very different things.
I mention this because of those commenting on this post on OpenSSL’s “constant-time” calculations, designed to avoid revealing secrets due to variations in compute time. The major comment is that it’s hard to do this perfectly in C. My response is that it’s hard to do this even in x86 machine code.
Consider registers, for example. Everyone knows that the 32-bit x86 was limited to 8 registers, while 64-bit expanded that to 16 registers. This isn’t actually true. The latest Intel processors have 168 registers. The name of the register in x86 code is really just a variable name, similar to how variables work in high-level languages.
So many registers are needed because the processor has 300 instructions “in flight” at any point in time in various stages of execution. It rearranges these instructions, executing them out-of-order. Everyone knows that processors can execute things slightly out-of-order, but that’s understated. Today’s processors are massively out-of-order.
Consider the traditional branch pair of a CMP (compare) followed by a JMPcc (conditional jump). While this is defined as two separate instructions as far as we humans are concerned, it’s now a single instruction as far as the processor is concerned.
Consider the “xor eax, eax” instruction, which is how we’ve traditionally cleared registers. This is never executed as an instruction, but just marks “eax” as no longer used, so that the next time an instructions needs the register, to allocate a new (zeroed) register from that pool of 168 registers.
Consider “mov eax, ebx”. Again, this doesn’t do anything, except rename the register as far as the processor is concerned, so that from this point on, what was referred to as ebx is now eax.
The processor has to stop and wait 5 clock cycles to read something from L1 cache, 12 cycles for L2 cache, or 30 cycles for L3 cache. But because the processor is massively out-of-order, I can continue executing instructions in the future that don’t depend upon this memory read. This includes other memory reads. Inside the CPU, the results always appear as if the processor executed everything in-order, but outside the CPU, things happen in strange order.
This means any attempt to get smooth, predictable execution out of the processor is very difficult. That means “side-channel” attacks on x86 leaking software crypto secrets may always be with us.
One solution to these problems is the CMOV, “conditional move”, instruction. It’s like a normal “MOV” instruction, but succeeds or fails based on condition flags. It can be used in some cases to replace branches, which makes pipelined code more efficient in some cases. Currently, it takes constant time. When moving from memory, it still waits for data to arrive, even when it knows it’s going to throw it away. As Linus Torvalds famously pointed out, CMOV doesn’t always speed up code. However, that’s not the point here — it does make code execution time more predictable. But, at the same time, Intel can arbitrarily change the behavior on future processors, making it less predictable.
The upshot is this: Intel’s x86 is a high-level language. Coding everything up according to Agner Fog’s instruction timings still won’t produce the predictable, constant-time code you are looking for. There may be some solutions, like using CMOV, but it will take research.
The BIOS boots a computer and helps load the operating system. By infecting this core software, which operates below antivirus and other security products and therefore is not usually scanned by them, spies can plant malware that remains live and undetected even if the computer’s operating system were wiped and re-installed.
Although most BIOS have protections to prevent unauthorized modifications, the researchers were able to bypass these to reflash the BIOS and implant their malicious code.
Because many BIOS share some of the same code, they were able to uncover vulnerabilities in 80 percent of the PCs they examined, including ones from Dell, Lenovo and HP. The vulnerabilities, which they’re calling incursion vulnerabilities, were so easy to find that they wrote a script to automate the process and eventually stopped counting the vulns it uncovered because there were too many.
Kallenberg said an attacker would need to already have remote access to a compromised computer in order to execute the implant and elevate privileges on the machine through the hardware. Their exploit turns down existing protections in place to prevent re-flashing of the firmware, enabling the implant to be inserted and executed.
The devious part of their exploit is that they’ve found a way to insert their agent into System Management Mode, which is used by firmware and runs separately from the operating system, managing various hardware controls. System Management Mode also has access to memory, which puts supposedly secure operating systems such as Tails in the line of fire of the implant.
From the Register:
“Because almost no one patches their BIOSes, almost every BIOS in the wild is affected by at least one vulnerability, and can be infected,” Kopvah says.
“The high amount of code reuse across UEFI BIOSes means that BIOS infection can be automatic and reliable.
“The point is less about how vendors don’t fix the problems, and more how the vendors’ fixes are going un-applied by users, corporations, and governments.”
Though such “voodoo” hacking will likely remain a tool in the arsenal of intelligence and military agencies, it’s getting easier, Kallenberg and Kovah believe. This is in part due to the widespread adoption of UEFI, a framework that makes it easier for the vendors along the manufacturing chain to add modules and tinker with the code. That’s proven useful for the good guys, but also made it simpler for researchers to inspect the BIOS, find holes and create tools that find problems, allowing Kallenberg and Kovah to show off exploits across different PCs. In the demo to FORBES, an HP PC was used to carry out an attack on an ASUS machine. Kovah claimed that in tests across different PCs, he was able to find and exploit BIOS vulnerabilities across 80 per cent of machines he had access to and he could find flaws in the remaining 10 per cent.
“There are protections in place that are supposed to prevent you from flashing the BIOS and we’ve essentially automated a way to find vulnerabilities in this process to allow us to bypass them. It turns out bypassing the protections is pretty easy as well,” added Kallenberg.
The NSA has a term for vulnerabilities it think are exclusive to it: NOBUS, for “nobody but us.” Turns out that NOBUS is a flawed concept. As I keep saying: “Today’s top-secret programs become tomorrow’s PhD theses and the next day’s hacker tools.” By continuing to exploit these vulnerabilities rather than fixing them, the NSA is keeping us all vulnerable.
New research: Max Abrahms and Philip B.K. Potter, “Explaining Terrorism: Leadership Deficits and Militant Group Tactics,” International Organizations.
Abstract: Certain types of militant groups — those suffering from leadership deficits — are more likely to attack civilians. Their leadership deficits exacerbate the principal-agent problem between leaders and foot soldiers, who have stronger incentives to harm civilians. We establish the validity of this proposition with a tripartite research strategy that balances generalizability and identification. First, we demonstrate in a sample of militant organizations operating in the Middle East and North Africa that those lacking centralized leadership are prone to targeting civilians. Second, we show that when the leaderships of militant groups are degraded from drone strikes in the Afghanistan-Pakistan tribal regions, the selectivity of organizational violence plummets. Third, we elucidate the mechanism with a detailed case study of the al-Aqsa Martyrs Brigade, a Palestinian group that turned to terrorism during the Second Intifada because pressure on the leadership allowed low-level members to act on their preexisting incentives to attack civilians. These findings indicate that a lack of principal control is an important, underappreciated cause of militant group violence against civilians.
Abstract: Research on security warnings consistently points to habituation as a key reason why users ignore security warnings. However, because habituation as a mental state is difficult to observe, previous research has examined habituation indirectly by observing its influence on security behaviors. This study addresses this gap by using functional magnetic resonance imaging (fMRI) to open the “black box” of the brain to observe habituation as it develops in response to security warnings. Our results show a dramatic drop in the visual processing centers of the brain after only the second exposure to a warning, with further decreases with subsequent exposures. To combat the problem of habituation, we designed a polymorphic warning that changes its appearance. We show in two separate experiments using fMRI and mouse cursor tracking that our polymorphic warning is substantially more resistant to habituation than conventional warnings. Together, our neurophysiological findings illustrate the considerable influence of human biology on users’ habituation to security warnings.
The Times Higher Education World University Rankings 2014-2015
|2015 rank||2014 rank||Institution|
|1||1||Harvard University (US)|
|2||4||University of Cambridge (UK)|
|3||5||University of Oxford (UK)|
|4||2||Massachusetts Institute of Technology (US)|
|5||3||Stanford University (US)|
|6||6||University of California, Berkeley (US)|
|7||7||Princeton University (US)|
|8||8||Yale University (US)|
|9||9||California Institute of Technology (US)|
|10||12||Columbia University (US)|
Even as new services like Spotify and Netflix gain traction, people are still flocking to file-sharing networks in their millions. These days people are increasingly likely to get a warning letter in the post advising them to mend their ways or face bigger trouble, but tougher approaches still exist.
While being targeted by a copyright troll must be a pretty miserable experience, being arrested has to be a lot worse. It only happens rarely and when it does it tends to affect the tech savvy 18-to-35s who grew up with the social norm of sharing files online. On occasion, however, it happens to those much older.
In 2011, a 58-year-old grandmother from Scotland was arrested and eventually sentenced to three years probation for sharing files online. However a new case in Europe has cast that earlier one into the shadows.
According to police in Romania, a 63-year-old woman has just been arrested for sharing files using BitTorrent. The raid took place in Cluj-Napoca (commonly known as Cluj), the second most populous city in Romania after the capital Bucharest.
“Following investigations by the economic crime investigation, police in Cluj…prosecuted a 63-year-old woman,” a police statement reads.
“This investigation was about the offense of making content available to the public, including via the Internet or other computer networks so that the public can access it anywhere and at any time individually chosen.”
Local police say their research revealed that the woman had been making available significant quantities of movies, music and other content without the necessary permission from rights holders. While that doesn’t sound out of the ordinary, the country doesn’t have much of a record for this kind of action. In fact, many torrent sites themselves operate out of Romania trouble free.
A source familiar with the copyright and enforcement scene in Romania told TorrentFreak that while it is indeed unusual for someone so old to be prosecuted for file-sharing, in Romania the prosecution of file-sharers of any age is “very very rare.”
“The police are doing this on their own? Never,” he said. “They only follow [pressure from] companies.”
The suggestion that complaints from rightsholders prompted the arrest is not an unusual one and Romanian media notes that entertainment company involvement in the case will continue as potential damages claims are assessed.
The lady at the center of this Romanian case is quite possibly the oldest file-sharer to be prosecuted anywhere in the world. The case that featured the youngest alleged pirate – just 9-years-old – became infamous following the confiscation of a Winnie-the-Pooh laptop.
September last year the Digital Citizens Alliance and NetNames released a report that looked into the business models of “shadowy” file-storage sites.
Titled “Behind The Cyberlocker Door: A Report How Shadowy Cyberlockers Use Credit Card Companies to Make Millions,” the report offers insight into the money streams that end up at these alleged pirate sites.
The research claims that the sites in question are mostly used for copyright infringement. But while there are indeed many shadowy hosting services, many were surprised to see the Kim Dotcom-founded Mega.co.nz on there.
For entertainment industry groups the report offered an opportunity to put pressure on Visa and MasterCard. In doing so they received support from U.S. Senator Patrick Leahy, who was also the lead sponsor of the defunct controversial Protect IP Act (PIPA).
Senator Leahy wrote a letter to the credit card companies claiming that the sites mentioned in the report have “no legitimate purpose or activity,” hoping they would cut their connections to the mentioned sites.
Visa and MasterCard took these concerns to heart and pressed PayPal to cut off its services to Mega, which eventually happened late last month. Interestingly, PayPal cited Mega’s end-to-end-encryption as one of the key problems, as that would make it harder to see what files users store.
The PayPal ban has been a huge blow for Mega, both reputation-wise and financially. And the realization that the controversial NetNames report is one of the main facilitators of the problems is all the more frustrating.
TorrentFreak spoke with CEO Graham Gaylard, who previously characterized the report as “grossly untrue and highly defamatory,” to discuss whether Mega still intends to take steps against the UK-based NetNames for their accusations.
Initially, taking legal action against NetNames for defamation was difficult, as UK law requires the complaining party to show economic damage. However, after the PayPal ban this shouldn’t be hard to do.
Gaylard is traveling through Europe at the moment and he notes that possible repercussions against the damaging report are high on the agenda.
“Yes, I am here to see Mega’s London-based legal counsel to discuss the next steps in progressing the NetNames’ response,” Gaylard informs TF.
Mega’s CEO couldn’t release any details on a possible defamation lawsuit, but he stressed that his company will fiercely defend itself against smear campaigns.
“Mega has been operating, and continues to operate a completely legitimate and transparent business. Unfortunately now, with the blatant, obvious, political pressure and industry lobbying against Mega, Mega needs to defend itself and will now cease taking a passive stance,” Gaylard says.
According to the CEO Mega is running a perfectly legal business. The allegation that it’s a piracy haven is completely fabricated. Like any other storage provider, there is copyrighted content on Mega’s servers, but that’s a tiny fraction of the total stored.
To illustrate this, Gaylard mentions that they only receive a few hundred takedown notices per month. In addition, he notes more than 99.7% of the 18 million files that are uploaded per day are smaller than 20MB in size, not enough to share a movie or TV-show.
These statistics are certainly not the hallmark of a service with “no legitimate purpose or activity,” as was claimed.
While the PayPal ban is a major setback, Mega is still doing well in terms of growth. They have 15 million registered customers across 200 countries, and hundreds of thousands of new users join every month.
New research: Geotagging One Hundred Million Twitter Accounts with Total Variation Minimization,” by Ryan Compton, David Jurgens, and David Allen.
Abstract: Geographically annotated social media is extremely valuable for modern information retrieval. However, when researchers can only access publicly-visible data, one quickly finds that social media users rarely publish location information. In this work, we provide a method which can geolocate the overwhelming majority of active Twitter users, independent of their location sharing preferences, using only publicly-visible Twitter data.
Our method infers an unknown user’s location by examining their friend’s locations. We frame the geotagging problem as an optimization over a social network with a total variation-based objective and provide a scalable and distributed algorithm for its solution. Furthermore, we show how a robust estimate of the geographic dispersion of each user’s ego network can be used as a per-user accuracy measure which is effective at removing outlying errors.
Leave-many-out evaluation shows that our method is able to infer location for 101,846,236 Twitter users at a median error of 6.38 km, allowing us to geotag over 80% of public tweets.