Abstract: September 11 created a natural experiment that enables us to track the psychological effects of a large-scale terror event over time. The archival data came from 8,070 participants of 10 ABC and CBS News polls collected from September 2001 until September 2006. Six questions investigated emotional, behavioral, and cognitive responses to the events of September 11 over a five-year period. We found that heightened responses after September 11 dissipated and reached a plateau at various points in time over a five-year period. We also found that emotional, cognitive, and behavioral reactions were moderated by age, sex, political affiliation, and proximity to the attack. Both emotional and behavioral responses returned to a normal state after one year, whereas cognitively-based perceptions of risk were still diminishing as late as September 2006. These results provide insight into how individuals will perceive and respond to future similar attacks.
Posts tagged ‘research’
There’s a new paper on a low-cost TEMPEST attack against PC cryptography:
We demonstrate the extraction of secret decryption keys from laptop computers, by nonintrusively measuring electromagnetic emanations for a few seconds from a distance of 50 cm. The attack can be executed using cheap and readily-available equipment: a consumer-grade radio receiver or a Software Defined Radio USB dongle. The setup is compact and can operate untethered; it can be easily concealed, e.g., inside pita bread. Common laptops, and popular implementations of RSA and ElGamal encryptions, are vulnerable to this attack, including those that implement the decryption using modern exponentiation algorithms such as sliding-window, or even its side-channel resistant variant, fixed-window (m-ary) exponentiation.
We successfully extracted keys from laptops of various models running GnuPG (popular open source encryption software, implementing the OpenPGP standard), within a few seconds. The attack sends a few carefully-crafted ciphertexts, and when these are decrypted by the target computer, they trigger the occurrence of specially-structured values inside the decryption software. These special values cause observable fluctuations in the electromagnetic field surrounding the laptop, in a way that depends on the pattern of key bits (specifically, the key-bits window in the exponentiation routine). The secret key can be deduced from these fluctuations, through signal processing and cryptanalysis.
Researchers at Tel Aviv University and Israel’s Technion research institute have developed a new palm-sized device that can wirelessly steal data from a nearby laptop based on the radio waves leaked by its processor’s power use. Their spy bug, built for less than $300, is designed to allow anyone to “listen” to the accidental radio emanations of a computer’s electronics from 19 inches away and derive the user’s secret decryption keys, enabling the attacker to read their encrypted communications. And that device, described in a paper they’re presenting at the Workshop on Cryptographic Hardware and Embedded Systems in September, is both cheaper and more compact than similar attacks from the past — so small, in fact, that the Israeli researchers demonstrated it can fit inside a piece of pita bread.
Cybercriminals have long relied on compromised Web sites to host malicious software for use in drive-by download attacks, but at least one crime gang is taking it a step further: New research shows that crooks spreading the Dyre malware for use in cyberheists are leveraging hacked wireless routers to deliver their password-stealing crimeware.
According to a recent in-depth report from Symantec, Dyre is a highly developed piece of malware, capable of hijacking all three major web browsers and intercepting internet banking sessions in order to harvest the victim’s credentials and send them to the attackers. Dyre is often used to download additional malware on to the victim’s computer, and in many cases the victim machine is added to a botnet which is then used to send out thousands of spam emails in order to spread the threat.
Recently, researchers at the Fujitsu Security Operations Center in Warrington, UK began tracking Upatre being served from hundreds of compromised home routers — particularly routers powered by MikroTik and Ubiquiti’s AirOS.
“We have seen literally hundreds of wireless access points, and routers connected in relation to this botnet, usually AirOS,” said Bryan Campbell, lead threat intelligence analyst at Fujitsu. “The consistency in which the botnet is communicating with compromised routers in relation to both distribution and communication leads us to believe known vulnerabilities are being exploited in the firmware which allows this to occur.”
Campbell said it’s not clear why so many routers appear to be implicated in the botnet. Perhaps the attackers are merely exploiting routers with default credentials (e.g., “ubnt” for both username and password on most Ubiquiti AirOS routers). Fujitsu also found a disturbing number of the systems in the botnet had the port for telnet connections wide open.
In January 2015, KrebsOnSecurity broke the news that the botnet used to attack and briefly knock offline Microsoft’s Xbox and Sony Playstation’s networks relied entirely on hacked routers, all of which appeared to have been compromised remotely via telnet.
Whether you use a router from Ubiquiti or any other manufacturer, if you haven’t changed the default credentials on the device, it’s time to take care of that. If you don’t know whether you’ve changed the default administrative credentials for your wired or wireless router, you probably haven’t. Pop on over to routerpasswords.com and look up the make and model of your router.
To see whether your credentials are the default, you’ll need to open up a browser and enter the numeric address of your router’s administration page. For most routers, this will be 192.168.1.1 or 192.168.0.1. This page lists the default internal address for most routers. If you have no luck there, here’s a decent tutorial that should help most users find this address. And check out my Tools for a Safer PC primer for more tips on how to beef up the security of your router and your Web browser.
Im sure most of you are familiar with the EICAR (European Institute for Computer Antivirus Research) test file. Your anti-virus application should detect the EICAR test file the same way it detects malicious files. But it is a test file, so of course, the EICAR file is not malicious.
If you have doubts that an anti-virus application is working correctly, you use the EICAR test file. If the file is not detected, there is a problem.
If you have doubts that anti-virus alerts are properly delivered to your SIEM, you use the EICAR test file.
There are many examples where the EICAR test file comes in handy.
But using the EICAR test file has become more difficult over the years, because there are more and more security applications and devices that detect it. For example, downloading the EICAR test file in a corporate environment will often fail, because the anti-virus on your proxy will detect and block it.
Thats why I decided many years ago to create a program that writes the EICAR test file to disk when it is executed. The anti-virus program should not detect the EICAR test string inside my program (per the EICAR test file convention), but they should detect it when its written to disk. My program, EICARgen, worked fine for many years, but this has changed since a couple of years. Now many anti-virus programs detect EICARgen as a dropper (malware that writes its payload to disk).
I developed a new version: now when EICARgen is executed, nothing happens. It will only write the EICAR test file to disk when you pass it the proper argument: EICARgen write.
And now I come to the point of this diary entry. This new version of EICARgen is not only able to write the EICAR test file to disk, but also a couple of container files that contain the EICAR test file: a ZIP file, a PDF file and an Excel file. This is useful to test the settings of your anti-virus. For example, if your anti-virus is configured to scan the content of ZIP files, then you can use EICARgen to test this: EICARgen.exe zip eicar.zip.
I also have a video of EICARgen in action.
Please write a comment if you have other examples of file formats that you use when testing your anti-virus. Or if you have an idea for a file format to add to EICARgen.
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Anyway, седмицата беше пълна с всякакви весели неща:
– 7-8-9 човека, вързани по тунели, няколко и с BGP (които могат да се видят в моя nagios). Все още хората се добавят на ръка, Мариян е почнал да пише интерфейс (явно му е скучно).
– Пуснахме IPv6 в ISP-то, дооправихме някакви неща, повече – в status report-а. Две бързи интересни неща – IPv6 на alpha-та и nagios-а на ISP-то (user/pass guest/guest).
– По горната тема, трябва да пусна looking glass-ове. Приемам идеи за неща, които поддържат bird, cisco и quagga и не искат да им се прави черна магия, че да тръгнат (тоя от сайта на bird е тотална трагедия).
– Преди година и нещо, след една лекция на Боян Кроснов за инфраструктура, доста хора се заинтересуваха да видят истински datacenter. Два месеца след това бях направил една уговорка с Telepoint, и миналата седмица (никак не се забавих) в четвъртък и петък с групи от по 5-6 човека обиколихме и видяхме как изглежда и какви забавни неща има (например овлажнители на въздуха). По принцип е ориентирано към студенти, та ако има още интересуващи се, пишете ми, за да събера пак една-две групи.
(целим се в сряда като ден за посещение, щото тогава тествали генераторите…)
– Тъкмо привършва HackTUES, на което не можах да отида, заради курса и всичките останали забавления, жалко.
– И идва BurgasConf, следващата седмица. Мразя да пътувам, но съм обещал… Тази година няма да говоря там, имах идея за една лекция, но там ще се иска още много research.
With a net income of more than $1 billion Elsevier is one of the largest academic publishers in the world.
The company has the rights to many academic publications where scientists publish their latest breakthroughs. Most of these journals are locked behind paywalls, which makes it impossible for less fortunate researchers to access them.
Sci-Hub.org is one of the main sites that circumvents this artificial barrier. Founded by Alexandra Elbakyan, a researcher born and graduated in Kazakhstan, its main goal is to provide the less privileged with access to science and knowledge.
The service is nothing like the average pirate site. It wasn’t started to share the latest Hollywood blockbusters, but to gain access to critical knowledge that researchers require to do their work.
“When I was working on my research project, I found out that all research papers I needed for work were paywalled. I was a student in Kazakhstan at the time and our university was not subscribed to anything,” Alexandra tells TF.
After Googling for a while Alexandra stumbled upon various tools and services to bypass the paywalls. With her newly gained knowledge, she then started participating in online forums where other researchers requested papers.
When she noticed how grateful others were for the papers she shared, Alexandra decided to automate the process by developing software that could allow anyone to search for and access papers. That’s when Sci-Hub was born, back in 2011.
“The software immediately became popular among Russian researchers. There was no big idea behind the project, like ‘make all information free’ or something like that. We just needed to read all these papers to do our research,” Alexandra.
“Now, the goal is to collect all research papers ever published, and make them free,” she adds.
Of course Alexandra knew that the website could lead to legal trouble. In that regard, the lawsuit filed by Elsevier doesn’t come as a surprise. However, she is more than willing to fight for the right to access knowledge, as others did before her.
“Thanks to Elsevier’s lawsuit, I got past the point of no return. At this time I either have to prove we have the full right to do this or risk being executed like other ‘pirates’,” she says, naming Aaron Swartz as an example.
“If Elsevier manages to shut down our projects or force them into the darknet, that will demonstrate an important idea: that the public does not have the right to knowledge. We have to win over Elsevier and other publishers and show that what these commercial companies are doing is fundamentally wrong.”
The idea that a commercial outfit can exploit the work of researchers, who themselves are often not paid for their contributions, and hide it from large parts of the academic world, is something she does not accept.
“Everyone should have access to knowledge regardless of their income or affiliation. And that’s absolutely legal. Also the idea that knowledge can be a private property of some commercial company sounds absolutely weird to me.”
Most research institutions in Russia, in developing countries and even in the U.S. and Europe can’t afford expensive subscriptions. This means that they can’t access crucial research, including biomedical research such as cancer studies.
So aside from the public at large, Sci-Hub is also an essential tool for academics. In fact, some researchers use the site to access their own publications, because these are also locked behind a paywall.
“The funniest thing I was told multiple times by researchers is that they have to download their own published articles from Sci-Hub. Even authors do not have access to their own work,” Alexandra says.
Instead of seeing herself as the offender, Alexandra believes that the major academic publishers are the ones who are wrong.
“I think Elsevier’s business model is itself illegal,” she says, pointing to article 27 of the UN declaration on human rights which reads that “everyone has the right freely to participate in the cultural life of the community, to enjoy the arts and to share in scientific advancement and its benefits.”
The paywalls of Elsevier and other publishers violate this right, she believes. The same article 27 also allows authors to protect their works, but the publishers are not the ‘authors,’ they merely exploit the copyrights.
Alexandra insists that her website is legal and hopes that future changes in copyright law will reflect this. As for the Elsevier lawsuit, she’s not afraid to fight for her rights and already offers a public confession right here.
“I developed the Sci-Hub.org website where anyone can download paywalled research papers by request. Also I uploaded at least half of more than 41 million paywalled papers to the LibGen database and worked actively to create mirrors of it.
“I am not afraid to say this, because when you do the right thing, why should you hide it?” she concludes.
Note: Sci-Hub is temporarily using the sci-hub.club domain name. The .org will be operational again next week.
While some of the unit’s activities are focused on the claimed areas, JTRIG also appears to be intimately involved in traditional law enforcement areas and U.K.-specific activity, as previously unpublished documents demonstrate. An August 2009 JTRIG memo entitled “Operational Highlights” boasts of “GCHQ’s first serious crime effects operation” against a website that was identifying police informants and members of a witness protection program. Another operation investigated an Internet forum allegedly “used to facilitate and execute online fraud.” The document also describes GCHQ advice provided :to assist the UK negotiating team on climate change.”
Particularly revealing is a fascinating 42-page document from 2011 detailing JTRIG’s activities. It provides the most comprehensive and sweeping insight to date into the scope of this unit’s extreme methods. Entitled “Behavioral Science Support for JTRIG’s Effects and Online HUMINT [Human Intelligence] Operations,” it describes the types of targets on which the unit focuses, the psychological and behavioral research it commissions and exploits, and its future organizational aspirations. It is authored by a psychologist, Mandeep K. Dhami.
Among other things, the document lays out the tactics the agency uses to manipulate public opinion, its scientific and psychological research into how human thinking and behavior can be influenced, and the broad range of targets that are traditionally the province of law enforcement rather than intelligence agencies.
XOR DDOS Trojan Trouble
I have struggled over the past recent months with a clients environment becoming infected and reinfected with an XOR DDOS trojan. The disruption and reinfection rates were costly at times. The client in question is asmall business with limited resources. Since the systems would get reinfected, a baited system was eventually put into place to determine the incoming vector. It was not proven, but believed that ssh brute force was the incoming vector of attack. Once the attackerswere onto the server, a root kit trojan was used. A very intelligent one. I highly recommend that anyone that gets nabbed by this trojan or one like it reinstall your operating system as soon as possible and executemy prevention steps outlined below.
However, there are some circumstances that require mitigation before available resources can be used for reinstall/replacement and prevention measures. The client was in a situation where taking the system offline was not an immediate option. I placedsome really great links below.    Theyreview, analyze and fully confirm what wewere experiencing was the same. There were some minor differences. However, they never really offered a short term mitigation path to follow. Only somewhere in a comment on a forum (possibly one of the three articles below), did someone make a suggestion to change the file/directory attributes to assist in mitigation. It was only a suggestion with no further follow-up. Mitigation of this trojanwas difficult as it was intelligent enough to always restart when it was killed, which includedhelp from crontab entries every three minutes.”>The victim server was CentOS 6.5 system with a basic LAMP setup, that offered ssh and VSFTP services. Iptables was in use, but NOT SELinux. It is my untested claim that SELinux likely would have prevented this trojan from taking hold. I am not an SELinux user/expert so I was unable to take time to add it to this environment. “>/lib/libgcc4.so . This exe was perpetuatedvia cron”> /etc/crontab every three minutes. “>(*/3 * * * * root /etc/cron.hourly/udev.sh )
If crontab gets cleaned and an executable is still running, then the crontab will be repopulated on Friday night around midnight. “>/etc/init.d/* . “>ls -lrt /etc/init.d/* to discover some evidence. “>top utility, you can determine how many are running. If the startups are deleted, then more executables and startup scripts will be created and begin to run as well.
The malware itself was used as a DDOSagent. It took commands from a CC. The IP addresses it would communicate with were available from the strings output of the executable. When the malware agentwas called into action, the entire server and local pipe was saturated and consequently cut off from service.
The following steps were taken for mitigation. “>chattr command. “>/lib directories were helpful in preventing the malware from repopulating. I put together the following for loop script and added the following IP addresses to IP tables to drop all communication. The for loop consists of clean up of four running processes. “>PID”>kill command. “>for f in zyjuzaaame lcmowpgenr belmyowxlc aqewcdyppt
mv /etc/init.d/$f /tmp/ddos/
rm -f /etc/cron.hourly/udev.sh
rm -f /var/run/udev.pid
mv /lib/libgcc4.so /tmp/ddos/libgcc4.so.$f
chattr -R +i /lib
chattr -R +i /etc/init.d
“>IP Addresses to drop all traffic: “>18.104.22.168 “>Prevention
I now keep the immutable bit set on /lib on a clean system. It turn it off before patching and software installs, in the event the /lib directory is needed for updating.
I also recommend installing fail2ban and configuring it to watch many of your services. I have it currently watching apache logs, ssh, vsftp, webmail, etc. It really seems to be hitting the mark for prevention. There is a whitelist feature to ignore traffic from a given IP or IP range. This helps to keep the annoying customers from becoming a nag.
If you have experienced anything like the above, then please feel free to share. This analysis is only scratching the surface. The links below do a much deeper dive on this piece of malware.
ISC Handler on Duty
(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.
Netflix, Hulu and a host of other content streaming services block non-U.S. users from viewing their content. As a result, many people residing in or traveling outside of the United States seek to circumvent such restrictions by using services that advertise “free” and “open” Web proxies capable of routing browser traffic through U.S.-based computers and networks. Perhaps unsurprisingly, new research suggests that most of these “free” offerings are anything but, and actively seek to weaken browser security and privacy.
The data comes from Austrian researcher and teacher Christian Haschek, who published a simple script to check 443 open Web proxies (no, that number was not accidental). His script tries to see if a given proxy allows encrypted browser traffic (https://), and whether the proxy tries to modify site content or inject any content into the user’s browser session, such as ads or malicious scripts.
Haschek found that 79 percent of the proxies he tested forced users to load pages in unencrypted (http://) mode, meaning the owners of those proxies could see all of the traffic in plain text.
“It could be because they want you to use http so they can analyze your traffic and steal your logins,” Haschek said. “If I’m a good guy setting up a server so that people can use it to be secure and anonymous, I’m going to allow people to use https. But what is my motive if I tell users http only?”
Haschek’s research also revealed that slightly more than 16 percent of the proxy servers were actively modifying static HTML pages to inject ads.
Virtual private networks (VPNs) allow users to tunnel their encrypted traffic to different countries, but increasingly online content providers are blocking popular VPN services as well. Tor offers users the ability to encrypt and tunnel traffic for free, but in my experience the service isn’t reliably fast enough to stream video.
Haschek suggests that users who wish to take advantage of open proxies pick ones that allow https traffic. He’s created and posted online a free tool that allows anyone to test whether a given proxy permits encrypted Web traffic, as well as whether the proxy truly hides the user’s real Internet address. This blog post explains more about his research methodology and script.
Users who wish to take advantage of open proxies also should consider doing so using a Live CD or virtual machine setup that makes it easy to reset the system to a clean installation after each use. I rely on the free VirtualBox platform to run multiple virtual machines, a handful of which I use to do much of my regular browsing, tweeting, emailing and other things that can lead sometimes to malicious links, scripts, etc.
I’ll most likely revisit setting up your own VirtualBox installation in a future post, but this tutorial offers a fairly easy-to-follow primer on how to run a Live CD installation of a Linux distribution of your choosing on top of VirtualBox.
The Dawn of Online Music Piracy
By 1994, the development of the first mp3 encoder was complete. Working at an audio research laboratory at Germany’s state-funded Fraunhofer Institute, engineers had labored for seven years and spent millions of dollars to develop a functioning prototype.
The encoder was marvelous—by exploiting inherent flaws in the human ear, it could reduce the size of compact disc audio by more than 90%, with minimal losses in quality. But Fraunhofer had been outmaneuvered in the marketplace, and couldn’t generate sales.
In desperation, they decided to distribute their encoder for free. They began by handing out floppy disks at trade shows and conferences. Soon, distribution moved to the Internet, with a limited-functionality DOS-based encoder posted on Fraunhofer’s FTP sites. The encoder was supposed to produce only low-bitrate files, and stop working after 20 uses. Quickly, it was cracked.
By late 1995, USENET was awash with pirated music files. Most of these were simple demonstrations of the technology, not full songs. Modern conveniences make it hard to remember the limitations of media distribution of the time; bandwidth meant 28,800 bits per second over a screeching telephone line, and compressing an mp3 from a CD meant a dedicated hour of CPU resources, accompanied by the buzz of a whirring fan.
The underground pirates of the Scene first adopted the technology in August of 1996. The pioneering group was Compress ‘Da Audio (CDA); their first release was Metallica’s “Until It Sleeps.” The full song was stored as a RAR file across four 3.5” floppy disk drives. These disks were then sent through the mail.
By late August, the rival Digital Audio Crew (DAC) had moved into the space; they posted an mp3-ripping tutorial to USENET, along with a direct link to Fraunhofer’s FTP site, accompanied by the serial numbers needed to unlock the encoder.
By the start of 1997, piracy had moved from floppy disks to campus servers, and processing power had doubled. Scene groups started releasing whole albums, not just individual singles. The files were no longer distributed through the postal service, but instead through IRC networks, FTP sites and even HTML links.
The Scene celebrated a “0-day” mentality—one gained notoriety by being the first to post pirated material to the Net. With music, that meant getting inside the retail industry’s supply chain.
The pioneering Scene group Rabid Neurosis (RNS) began infiltrating record stores, exploiting offset international release dates, and recruiting music journalists and commercial radio DJs. Music became available on the Internet weeks, sometimes months, before it was due in stores. In time, RNS became the dominant player, sourcing thousands of pre-release albums from Dell Glover and Tony Dockery, two workers at a North Carolina CD manufacturing plant.
A generation came of age in that IRC underground—for many users it was their formative experience online. Included were Shawn Fanning and Sean Parker, who’d met in an chat channel, where they’d shared their frustrations with the inefficiencies of late-90s file-swapping. Fanning, 18, wrote 80,000 lines of code, for a new peer-to-peer platform he called Napster. Parker, 19, was deputized to promote it. In June of 1999, the software débuted.
The golden age of online piracy had begun.
About The Author
Stephen Witt is a journalist from Brooklyn, New York.
He’s the author of “How Music Got Free,” a well-researched book about the rise of music piracy and the key players that contributed to the early success of online file-sharing.
A database supposedly from a sample of information stolen in the much publicized hack at the Office of Personnel Management (OPM) has been making the rounds in the cybercrime underground, with some ne’er-do-wells even offering to sell it as part of a larger package. But a review of the information made available as a teaser indicates that the database is instead a list of users stolen from a different government agency — Unicor.gov, also known as Federal Prison Industries.
Earlier this week, miscreants who frequent the Hell cybercrime forum (a “Deep Web” site reachable only via the Tor network) began passing around a text file that contained more than 23,000 records which appeared to be a user database populated exclusively by user accounts with dot-gov email addresses. I thought it rather unlikely that the file had anything to do with the OPM hack, which was widely attributed to Chinese hackers who are typically interested in espionage — not selling the data they steal on open-air markets.
As discussed in my Oct. 2014 post, How to Tell Data Leaks from Publicity Stunts, there are several simple techniques that often can be used to tell whether a given data set is what it claims to be. One method involves sampling email addresses from the leaked/hacked database and then using them in an attempt to create new accounts at the site in question. In most cases, online sites and services will allow only one account per email address, so if a large, random sampling of email addresses from the database all come back as already registered at the site you suspect is the breached entity, then it’s a safe guess the data came from that entity.
How to know the identity of the organization from which the database was stolen? In most cases, database files list the users in the order in which they registered on the site. As a result, the email addresses and/or usernames for the first half-dozen or more users listed in the database are most often from the database administrators and/or site designers. When all of those initial addresses have the same top-level domain — in this case “unicor.gov” — it’s a good bet that’s your victim organization.
According to Wikipedia, UNICOR is a wholly owned United States government corporation created in 1934 that uses penal labor from the Federal Bureau of Prisons to produce goods and services. It is apparently restricted to selling its products and services to federal government agencies, although recently private companies gained some access to UNICOR workforce. For instance, companies can outsource call centers to UNICOR. Case in point: If you call UNICOR’s main number off-hours, the voicemail message states that during business hours your call may be handled by an inmate!
On Tuesday, I reached out to UNICOR to let them know that it appeared their user database — including hashed passwords and other information — was being traded on underground cybercrime forums. On Wednesday, I heard back from Marianne Cantwell, the public information officer for UNICOR. Cantwell said a review of the information suggests it is related to an incident in September 2013, when Federal Prison Industries discovered unauthorized access to its public Web site.
“Since that time, the website software has been replaced to improve security. Assessments by proper law enforcement authorities were conducted to determine the extent of the incident, at the time it was discovered,” said Cantwell, who confirmed the incident hadn’t been previously disclosed publicly. “Limited individuals were deemed to be potentially impacted, and notifications were made as a precautionary measure. Federal Prison Industries is sensitive to ensuring the security of its systems and will continue to monitor this issue.”
The “website software” in question was ColdFusion, a Web application platform owned by Adobe Systems Inc. Around that same time, hackers were running around breaking into a number of government and corporate Web sites and databases using ColdFusion vulnerabilities. In October 2013, I wrote about criminals who had used ColdFusion exploits to break into and steal the database from the National White Collar Crime Center (NW3C), a congressionally-funded non-profit organization that provides training, investigative support and research to agencies and entities involved in the prevention, investigation and prosecution of cybercrime.
There is no information to link the hack at UNICOR to the crooks behind the NW3C compromise, but it’s interesting to note that those responsible for the NW3C attack also had control over the now-defunct identity theft service ssndob[dot]ms. That service, which was advertised on cybercrime forums, was powered in part by a small but powerful collection of hacked computers exclusively at top data brokers, including LexisNexis, Dun & Bradstreet, and HireRight/Kroll.
Normally, I don’t cover vulnerabilities about which the user can do little or nothing to prevent, but two newly detailed flaws affecting hundreds of millions of Android, iOS and Apple products probably deserve special exceptions.
The first is a zero-day bug in iOS and OS X that allows the theft of both Keychain (Apple’s password management system) and app passwords. The flaw, first revealed in an academic paper (PDF) released by researchers from Indiana University, Peking University and the Georgia Institute of Technology, involves a vulnerability in Apple’s latest operating system versions that enable an app approved for download by the Apple Store to gain unauthorized access to other apps’ sensitive data.
“More specifically, we found that the inter-app interaction services, including the keychain…can be exploited…to steal such confidential information as the passwords for iCloud, email and bank, and the secret token of Evernote,” the researchers wrote.
The team said they tested their findings by circumventing the restrictive security checks of the Apple Store, and that their attack apps were approved by the App Store in January 2015. According to the researchers, more than 88 percent of apps were “completely exposed” to the attack.
News of the research was first reported by The Register, which reported that Apple was first notified in October 2014 and that in February 2015 the company asked researchers to hold off disclosure for six months.
“The team was able to raid banking credentials from Google Chrome on the latest Mac OS X 10.10.3, using a sandboxed app to steal the system’s keychain and secret iCloud tokens, and passwords from password vaults,” The Register wrote. “Google’s Chromium security team was more responsive and removed Keychain integration for Chrome noting that it could likely not be solved at the application level. AgileBits, owner of popular software 1Password, said it could not find a way to ward off the attacks or make the malware ‘work harder’ some four months after disclosure.”
A story at 9to5mac.com suggests the malware the researchers created to run their experiments can’t directly access existing keychain entries, but instead does so indirectly by forcing users to log in manually and then capturing those credentials in a newly-created entry.
“For now, the best advice would appear to be cautious in downloading apps from unknown developers – even from the iOS and Mac App Stores – and to be alert to any occasion where you are asked to login manually when that login is usually done by Keychain,” 9to5’s Ben Lovejoy writes.
SAMSUNG KEYBOARD FLAW
Separately, researchers at mobile security firm NowSecure disclosed they’d found a serious vulnerability in a third-party keyboard app that is pre-installed on more than 600 million Samsung mobile devices — including the recently released Galaxy S6 — that allows attackers to remotely access resources like GPS, camera and microphone, secretly install malicious apps, eavesdrop on incoming/outgoing messages or voice calls, and access pictures and text messages on vulnerable devices.
The vulnerability in this case resides with an app called Swift keyboard, which according to researcher Ryan Welton runs from a privilege account on Samsung devices. The flaw can be exploited if the attacker can control or compromise the network to which the device is connected, such as a wireless hotspot or local network.
“This means that the keyboard was signed with Samsung’s private signing key and runs in one of the most privileged contexts on the device, system user, which is a notch short of being root,” Welton wrote in a blog post about the flaw, which was first disclosed at Black Hat London on Tuesday, along the release of proof-of-concept code.
Welton said NowSecure alerted Samsung in November 2014, and that at the end of March Samsung reported a patch released to mobile carriers for Android 4.2 and newer, but requested an additional three months deferral for public disclosure. Google’s Android security team was alerted in December 2014.
“While Samsung began providing a patch to mobile network operators in early 2015, it is unknown if the carriers have provided the patch to the devices on their network,” Welton said. “In addition, it is difficult to determine how many mobile device users remain vulnerable, given the devices models and number of network operators globally.” NowSecure has released a list of Samsung devices indexed by carrier and their individual patch status.
Samsung issued a statement saying it takes emerging security threats very seriously.
“Samsung KNOX has the capability to update the security policy of the phones, over-the-air, to invalidate any potential vulnerabilities caused by this issue. The security policy updates will begin rolling out in a few days,” the company said. “In addition to the security policy update, we are also working with SwiftKey to address potential risks going forward.”
A spokesperson for Google said the company took steps to mitigate the issue with the release of Android 5.0 in November 2014.
“Although these are most accurately characterized as application level issues, back with Android 5.0, we took proactive measures to reduce the risk of the issues being exploited,” Google said in a statement emailed to KrebsOnSecurity. “For the longer term, we are also in the process of reaching out to developers to ensure they follow best practices for secure application development.”
SwiftKey released a statement emphasizing that the company only became aware of the problem this week, and that it does not affect its keyboard applications available on Google Play or Apple App Store. “We are doing everything we can to support our long-time partner Samsung in their efforts to resolve this important security issue,” SwiftKey said in a blog post.
New Annenberg survey results indicate that marketers are misrepresenting a large majority of Americans by claiming that Americas give out information about themselves as a tradeoff for benefits they receive. To the contrary, the survey reveals most Americans do not believe that ‘data for discounts’ is a square deal.
The findings also suggest, in contrast to other academics’ claims, that Americans’ willingness to provide personal information to marketers cannot be explained by the public’s poor knowledge of the ins and outs of digital commerce. In fact, people who know more about ways marketers can use their personal information are more likely rather than less likely to accept discounts in exchange for data when presented with a real-life scenario.
Our findings, instead, support a new explanation: a majority of Americans are resigned to giving up their data — and that is why many appear to be engaging in tradeoffs. Resignation occurs when a person believes an undesirable outcome is inevitable and feels powerless to stop it. Rather than feeling able to make choices, Americans believe it is futile to manage what companies can learn about them. Our study reveals that more than half do not want to lose control over their information but also believe this loss of control has already happened.
By misrepresenting the American people and championing the tradeoff argument, marketers give policymakers false justifications for allowing the collection and use of all kinds of consumer data often in ways that the public find objectionable. Moreover, the futility we found, combined with a broad public fear about what companies can do with the data, portends serious difficulties not just for individuals but also — over time — for the institution of consumer commerce.
Apple announced last week that its Swift programming language — a
currently fully proprietary software successor to Objective C — will
probably be partially released under an OSI-approved license eventually.
Apple explicitly stated though that such released software will not be
copylefted. (Apple’s pathological hatred of copyleft is reasonably well
documented.) Apple’s announcement remained completely silent on patents,
and we should expect the chosen non-copyleft license
will not contain a patent grant.
(I’ve explained at
great length in the past why software patents are a particularly dangerous
threat to programming language infrastructure.)
Apple’s dogged pursuit for non-copyleft replacements for copylefted
software is far from new. For example, Apple has worked to create
replacements for Samba so they need not ship Samba in OSX. But, their
anti-copyleft witch hunt goes back much further. It began
Stallman himself famously led the world’s first GPL enforcement effort
against NeXT, and Objective-C was liberated. For a time, NeXT and
Apple worked upstream with GCC to make Objective-C better for the
community. But, that whole time, Apple was carefully plotting its escape
from the copyleft world. Fortuitously, Apple eventually discovered a
technically brilliant (but sadly non-copylefted) research programming
language and compiler system called LLVM. Since then, Apple has sunk
millions of dollars into making LLVM better. On the surface, that seems
like a win for software freedom, until you look at the bigger picture:
their goal is to end copyleft compilers. Their goal is to pick and choose
when and how programming language software is liberated. Swift is not a
shining example of Apple joining us in software freedom; rather, it’s a
recent example of Apple’s long-term strategy to manipulate open source
— giving our community occasional software freedom on Apple’s own
terms. Apple gives us no bread but says
let them eat cake
Apple’s got PR talent. They understand that merely announcing the
possibility of liberating proprietary software gets press. They know that
few people will follow through and determine how it went. Meanwhile, the
standing story becomes:
Wait, didn’t Apple open source Swift. Already, that false soundbite’s grip strengthens, even though
the answer remains a resoundingly
No!. However, I suspect that
Apple will probably meet most
public pledges. We’ll likely see pieces of Swift 2.0 thrown over the
wall. But the best stuff will be kept proprietary. That’s already happening
with LLVM, anyway; Apple already ships a no-source-available fork of
Thus, Apple’s announcement incident hasn’t happened in a void. Apple
didn’t just discover open source after years of neutrality on the topic.
Apple’s move is calculated, which
industry pundits like O’Grady and Weinberg to ask hard questions (some of
which are similar to mine). Yet, Apple’s hype is so good, that
convince one trade association leader.
To me, Apple’s not-yet-executed move to liberate some of the Swift 2.0
code seems a tactical stunt to win over developers who currently prefer the
relatively more open nature of the Android/Linux platform. While nearly
all the Android userspace applications are proprietary, and GPL violations on
Android devices abound, at least the copyleft license of Linux itself
provides the opportunity to keep the core operating system of Android
liberated. No matter how much Swift code is released, such will never be
true with Apple.
I’m often pointing out
in my recent
talks how complex and treacherous the Open Source and Free Software
political climate became in the last decade. Here’s a great example: Apple
is a wily opponent, able to Open Source (the cooption of Free Software) to
manipulate the press and hoodwink the would-be spokespeople for Linux to
support them. Many of us software freedom advocates have predicted for
years that Free Software unfriendly companies like Apple would liberate
more and more code under non-copyleft licenses in an effort to create
walled gardens of seeming software freedom. I don’t revel in my past
accuracy of such predictions; rather, I feel simply the hefty weight of
Encrypting your Windows hard drives is trivially easy; choosing which program to use is annoyingly difficult. I still use Windows — yes, I know, don’t even start — and have intimate experience with this issue.
Historically, I used PGP Disk. I used it because I knew and trusted the designers. I even used it after Symantec bought the company. But big companies are always suspect, because there are a lot of ways for governments to manipulate them.
For Windows, the options are basically BitLocker, Symantec’s PGP Disk, and TrueCrypt. I choose TrueCrypt as the least bad of all the options.
But soon after that, despite the public audit of TrueCrypt, I bailed for BitLocker.
BitLocker is Microsoft’s native file encryption program. Yes, it’s from a big company. But it was designed by my colleague and friend Niels Ferguson, whom I trust. (Here’s Niels’s statement from 2006 on back doors.) It was a snap decision; much had changed since 2006. (Here I am in March speculating about an NSA back door in BitLocker.) Specifically, Microsoft made a bunch of changes in BitLocker for Windows 8, including removing something Niels designed called the “Elephant Diffuser.”
The Intercept’s Micah Lee recently recommended BitLocker and got a lot of pushback from the security community. Last week, he published more research and explanation about the trade-offs. It’s worth reading. Microsoft told him they removed the Elephant Diffuser for performance reasons. And I agree with his ultimate conclusion:
Based on what I know about BitLocker, I think it’s perfectly fine for average Windows users to rely on, which is especially convenient considering it comes with many PCs. If it ever turns out that Microsoft is willing to include a backdoor in a major feature of Windows, then we have much bigger problems than the choice of disk encryption software anyway.
Whatever you choose, if trusting a proprietary operating system not to be malicious doesn’t fit your threat model, maybe it’s time to switch to Linux.
Micah also nicely explains how TrueCrypt is becoming antiquated, and not keeping up with Microsoft’s file system changes.
Lately, I am liking an obscure program called BestCrypt, by a Finnish company called Jetico. Micah quotes me:
Considering Schneier has been outspoken for decades about the importance of open source cryptography, I asked if he recommends that other people use BestCrypt, even though it’s proprietary. “I do recommend BestCrypt,” Schneier told me, “because I have met people at the company and I have a good feeling about them. Of course I don’t know for sure; this business is all about trust. But right now, given what I know, I trust them.”
I know it’s not a great argument. But, again, I’m trying to find the least bad option. And in the end, you either have to write your own software or trust someone else to write it for you.
But, yes, this should be an easier decision.
I’ve been spending my evenings working on various tweaks and possible enhancements to the FotoForensics site. Some of these experiments have worked out really well, some have a few problems, and some are still in the “learning curve” phase.
This snapshot shows related clusters that formed after three days. The actual graph is interactive. I can click on any node to identify more information about it. I like the way it displays and it really helps identify various clusters of related data. However, this tool really isn’t practical for a public release because of two big problems.
I’m not the only person to notice this memory leak. It seems to impact newer versions of Firefox, Chrome, and Chromium, dating back more than a year. (Examples: #1, #2, #3.) I suspect that Safari and other WebKit browsers may have the same problem.
And before someone asks… Yes, I tried the latest-greatest versions of Chrome and Firefox. Both crash on Ubuntu 14.04 before they can load any pages. (These are unstable browser ports.) On Windows, they still have the memory leak. For right now, I’m only using this vis.js code on an old Chrome browser that predates the memory leak. Ideally I’d like an interactive web-based solution that can handle 10,000 nodes, but that doesn’t seem likely in the near future.
I’ve been spending some time trying to wrap my head around WebRTC. That’s the interactive web technology that permits video and audio sharing. My long-term goal is to configure a WebRTC server for FotoForensics, where I can share my browser window and conduct online training sessions for specific clients, research partners, and occasional guests. (This isn’t intended for the public FotoForensics server. This is more for the private servers that have more features and really requires training sessions.)
I’ve finally wrapped my head around the WebRTC, STUN, and TURN relationships that are required for enabling this technology. There are dozens of web pages with overviews and tutorials, but none of them are very good or detailed. And I still need to figure out how to do things like encryption. (Some docs say that the traffic is automagically encrypted, but I cannot find details about how this works.)
Installing, configuring, and deploying is another complex issue. While there are a few ready-to-go installation packages, I haven’t found them easy to customize. For example, I have my own login management system but I cannot figure out how to integrate it. I want to make sure users cannot create their own private chat rooms, but most code enables arbitrary room allocations. And I want to share either an app (browser) or a desktop and not a video camera, but I cannot figure out how to do that. In some cases, I may just want to share audio without video, or audio with a text-chat window. In other cases, I want users to be able to share their desktops with me so that I can help diagnose content. But I haven’t figured out how to do any of these either.
I have also played with external systems, like Google Hangouts, GoToMeeting, and WebEx. I like the speed, I like the flow, and I like the features like a text-based chat window and Hangout’s live annotations. But I don’t like the idea of sending anything related to my technologies through a third party; all communications should go direct from my server and my desktop computer to the other member’s computers. I want no dependencies on any external third-party services. Also, anything that requires installing special software as a plugin or an app is a show-stopper. I need to support a lot of different platforms, and requiring every user to install a plugin or app is not a platform independent solution.
Outside of the graphical arena, I’ve been looking more at the various users who attack my site or violate the terms of service. If I can identify trends, then I can address them and cut down on abuses.
Recently I noticed that some of these abusers are using cloud service providers. So, I decided to map out which services they use. I really expected them to be evenly distributed across the various cloud solutions, but that is definitely not the case.
Some of the biggest cloud providers, like CloudFlare, Rackspace, Softlayer, and Microsoft’s Azure, do not show up at all in my lists of abusive sources. I assume that this means that they are very good at policing their users. (Either that, or these services are too expensive for the riff raff.) The cloud services offered by Google and Amazon do not have many violators, but nearly all of their violators are associated with hostile network attacks. These are systems that are explicitly trying to compromise other online computers. And in the case of Google, they have a few hostile accounts that have been going at it for at least a few months. Either these cloud services have not noticed that their users are hostile, or do not care about stopping outbound attacks.
In contrast to Google and Amazon, Versaweb, GTT/nLayer, and a few others are mostly associated with proxies that are used to violate my terms of service. (I.e., porn uploaders.) This makes it really easy to identify and I can flag their content as potential violations. I should have a new autoban rule implemented in the near future.
I’m still trying to finish up and deploy a few other technologies. Some of these will better protect my site, while others will make the site more convenient for users and analysts. Whenever I deploy an improvement to the site, I end up learning something new, and that may lead to additional fun research topics. I am definitely looking forward to these behind-the-scenes updates and whatever surprises they may bring.
Liz: Here’s a guest post from Bernat at Wolfram, who has been putting the Wolfram Language on the Pi to work at a Smart Cities Hackathon in Barcelona. If you haven’t used the Wolfram Language before, this is a nice little glimpse into what it’s capable of: enjoy!
On Friday, February 20, I had the pleasure of giving a talk to a group of young and smart individuals enlisted to represent Barcelona in the Global Urban Datafest. For this hackathon, the organizers offered one Raspberry Pi platform per team and a variety of sensors to capture physical parameters. Their list of suggested project topics included data acquisition and actuation, monitoring and management, security transport and mobility, the environment, and more. The event lasted three days and was locally organized by Anna Calveras and Josep Paradells with the help of Universitat Politècnica de Catalunya, Barcelona’s City Council, iCity Project, Urbiotica, IBM, and Wolfram Research.
Early on, hackathon participants were oriented to the various tools available to aid them with development. I showed the hackathon participants that the Wolfram Language knows about thousands of real-world entities, and that everything in the language is a symbolic expression.
Then I explained how I used a Raspberry Pi to digest Friday’s bicycle data overnight. The microprocessor was set up to compute the total number of bicycles available in different cities every 10 minutes from 3:30–8:30am CET:
European cities showed a valley in the number of available bicycles at 8am when people cycled to work. Citizens from New York and Mexico City were found to head back home around 5am CET.
Essential for this hackathon was the new Wolfram Data Drop, an open service that makes it easy to accumulate data of any kind, from anywhere, which works great on the Pi while connected to the Wolfram Cloud. The following is a dataset that I created for Barcelona’s bike-sharing system. Every 3 minutes the total number of parked bicycles is added to a Databin:
One of the cool features of Data Drop is that you can directly analyze this data through Wolfram|Alpha:
Another dataset that I created using a Raspberry Pi monitored the pedestrian flow happening at the front door of my apartment. If any movement was detected by a PIR motion sensor, the RaspiCam would take a photo, and a new entry would be added into a databin:
This appears in the Data Drop cloud like this:
The result was this DateListPlot of cumulative numbers of movements detected:
Then I showed how it could be set up to monitor my home hall’s activity in regular periods of time:
Certainly, this opens up a new world of possibilities. For example, you can use Data Drop to combine data from specific events from different devices. This was exactly what one of the teams did. They set up a Twitter account with ServiceConnect to inform people of the current air pollution in “La Diagonal,” Barcelona’s most important avenue. Every 20 minutes they checked the latest values of 10 gas sensors, and then generated and tweeted a ListLinePlot with a map of the sensors:
Other smart city projects involved the use of the new Machine Learning capabilities available in Mathematica 10, such as FindFaces to estimate the number of individuals in a bar, or BarcodeRecognize for a universal citizen ID card project. For most of the participants, this was their first encounter with the Wolfram Language, and yet they made useful, functional prototypes in just 48 hours. So I can’t wait to see what they are capable of with just a bit more practice. I wish all of them tons of happy, smart coding!
If you haven’t participated in a hackathon yet, check out the Smart City App Hack. Also feel free to contact us for future events, and don’t forget to have a look at Create, Code, Deploy: Workshop for Hackathons if you missed it. Finally, if you are looking for a three-week-long hackathon, apply now to the Wolfram Innovation Summer School or the Wolfram Science Summer School.
The post A Smart Programming Language for a Smart Cities Hackathon appeared first on Raspberry Pi.
With a net income of more than $1 billion Elsevier is one of the largest academic publishers in the world.
Through its ScienceDirect portal the company offers access to millions of scientific articles spread out over 2,200 journals.
Most large universities have licenses to allow staff and students to use ScienceDirect freely, but for outsiders most of the top academic publications are behind an expensive paywall.
In common with other content behind paywalls, there are several specialized sites that allow the general public to download pirated copies of these academic works. The Library Genesis project for example, with libgen.org and bookfi.org, as well as the search portal sci-hub.org.
These sites are particularly popular in developing countries such as Iran, India and Indonesia where access to research is not as common. However, this unauthorized use is not welcomed by academic publishers.
According to Elsevier the company is losing revenue because of these sites, so in order to stem the tide the publisher has filed a complaint (pdf) at a New York federal court hoping to shut them down.
“Defendants are reproducing and distributing unauthorized copies of Elsevier’s copyrighted materials, unlawfully obtained from ScienceDirect, through Sci-Hub and through various websites affiliated with the Library Genesis Project,” the complaint reads.
“Specifically, Defendants utilize their websites located at sci-hub.org and at the Libgen Domains to operate an international network of piracy and copyright infringement by circumventing legal and authorized means of access to the ScienceDirect database,” it adds.
According to Elsevier, the websites access articles by using unlawfully obtained student or faculty access credentials. The articles are then added to the “pirate” library, backed up on their own servers.
Through the lawsuit the publisher hopes to obtain an injunction against the site’s operators, search engines, domain registrars and hosting companies, to take them offline as soon as possible.
In addition, Elsevier is requesting compensation for its losses, which could run into the millions.
Tom Allen, President of the Association of American Publishers (AAP), informs TF that websites such as Libgen pose a threat to the quality of scientific publications, as well as the public health.
“Scholarly publishers work to ensure the accuracy of the scientific record by issuing corrections and revisions to research findings as needed; Libgen typically does not,” Allen says.
“As a result, its repository of illegally obtained content poses a threat to both quality journal publishing and to public health and safety.”
The court has yet to decide whether the injunctions should be granted, but considering outcomes in recent piracy cases there’s a good chance this will happen. For the time being, however, the Libgen and Sci-hub websites remain online.
This is interesting research: “How Near-Miss Events Amplify or Attenuate Risky Decision Making,” Catherine H. Tinsley, Robin L. Dillon, and Matthew A. Cronin.
In the aftermath of many natural and man-made disasters, people often wonder why those affected were underprepared, especially when the disaster was the result of known or regularly occurring hazards (e.g., hurricanes). We study one contributing factor: prior near-miss experiences. Near misses are events that have some nontrivial expectation of ending in disaster but, by chance, do not. We demonstrate that when near misses are interpreted as disasters that did not occur, people illegitimately underestimate the danger of subsequent hazardous situations and make riskier decisions (e.g., choosing not to engage in mitigation activities for the potential hazard). On the other hand, if near misses can be recognized and interpreted as disasters that almost happened, this will counter the basic “near-miss” effect and encourage more mitigation. We illustrate the robustness of this pattern across populations with varying levels of real expertise with hazards and different hazard contexts (household evacuation for a hurricane, Caribbean cruises during hurricane season, and deep-water oil drilling). We conclude with ideas to help people manage and communicate about risk.
While error level analysis (ELA) seems like a simple enough concept, budding analysts need to understand what the algorithm does, how it works, and how to apply it. By itself, ELA highlights the various compression level potentials across an image (analogous to adding dye to a petri dish). However, the analyst needs to know what to look for in the results. There’s the straightforward “significantly different” coloring, the more subtle chrominance separation (rainbowing), and other artifacts that alter the compression rate across the image.
To help with this learning curve, I developed tutorials and challenges. The tutorials describe how the algorithm works and the challenges allow people to test their knowledge. Since different teaching methods work better for different people, it is good to offer a variety of training methods.
I recently ran the stats on the FotoForensics tutorials. I checked them by weekly and monthly distributions and the results were consistent: about a third of visitors to the site (35% average) visit the tutorials page. However, only about a tenth of them actually spend more than a few seconds on the tutorials pages in a given week. The challenges average about 7%, but those users appear to work on at least one challenge puzzle.
I also looked for longer trends. A solid third (34%) of unique network addresses have spent time with the tutorials and/or challenges in the previous year. (I keep thinking: a free site where a third of the users are actually reading the training materials? WOW!)
I even see people applying what they learned. FotoForensics has been very popular with the Reddit community. A few years ago, people gave a lot of bad interpretations (e.g., “white means modified” or “color means fake” or “ELA doesn’t work”). However, those ~30% of users who took the time to learn have become a dominant force at Reddit. When someone posts a link to FotoForensics, it is usually followed by someone asking what it means, and someone else giving an intelligent answer.
On one hand, this tells me that the tutorials and challenges are easy enough for users to find on the site. And about 3 in 10 users are interested enough to take the time to learn how it works. (I am open to suggestions for other possible training options that could engage with more of the other 70%.)
Unfortunately, I still see people misapplying the technology or giving really bad advice on how it works.
It’s all about compression
ELA does one thing: it quantifies the lossy compression error potential over the image. It returns a map that shows the compression level over the image. It doesn’t return a numerical value (“7″) or summary (“green” or “true”) because different types of alterations generate different compression level signatures. For example, if a picture is 95% unaltered, then would you call it real or fake? With a map of the picture, you can identify the abnormal area.
Over at Reddit, a tom_beale posted to “mildly infuriating” some sidewalk covers that were put back wrong. User Afterfx21 “fixed it“. The compression map generated by ELA makes it easy to identify how it was “fixed”.
But let’s go back a moment and talk about compression…
JPEG is based on a lossy compression system. By “lossy”, we mean that the decompressed data does not look exactly like the pre-compressed data. What comes out is similar, but not exactly like what went in. Since there’s a little difference, it looses quality. Even saving a JPEG at “100%” will result in a little data loss; what most tools call “100%” is actually closer to 99%. The purpose of the lossy compression is to make as many repetitive zeros and small values as possible in the encoded sequence, while remaining visually similar to the source image. More repitition leads to better compression.
The lossy compression works by quantizing the values; effectively turning a smooth curve into stair steps. For example the quantization value “3” would make the values “40 20 10 5 1″ become “13 6 3 1 0″. JPEG uses integer math so fractions after dividing by 3 are lost.
To restore the sequence, the values are multiplied by the quantization value: “39 18 9 3 0″. Each of these decoded values are close enough to the source values. When talking about pixels, the human eye is unlikely to notice any difference. (The actual JPEG encoding method is a little more complicated and includes 64 quantization values as well as some other compression steps. For much more detail about JPEG encoding, see JPEG Bitstream Bytes.)
Additional JPEG compression loss
If we just use one quantization value and repeatedly cycle between encoding and decoding, then the first encoding will cause data loss but the remainder will not.
Encoding “40 20 10 5 1″ with quantizer “3” generates “13 6 3 1 0″. (Encoding is a division with integer math.)
Decoding “13 6 3 1 0″ with quantizer “3” generates “39 18 9 3 0″. (Decoding is a multiplication.)
Encoding “39 18 9 3 0″ with “3” generates “13 6 3 1 0″. (Same value)
Decoding “13 6 3 1 0″ with quantizer “3” generates “39 18 9 3 0″. (Same value)
If JPEG encoded RGB values, then this would be it. The first encoding would generates a little loss but repeated encoding/decoding cycles would not. Unfortunately, JPEG does not encode RGB values. Instead, it first converts the values from RGB to YUV (an alternate color space). This conversion is lossy and causes values to shift a little. This means two things. First, JPEG cannot store true 24-bit color. Second, the values may shift a little between the first decoding and second encoding steps, so the next encoding may result in values that are a little different.
But JPEG doesn’t stop there. It also converts the colors from the 8×8 grid pixel space to an 8×8 frequency space. This conversion uses a discrete cosine transform (DCT). When you see the word “cosine” you should be thinking “floating point values”. Except that JPEG does everything with integers so fractions get truncated. Simply repeatedly encoding and decoding the DCT values with integer math will result in constant degradation. When combined with the quantization step, it results in significant degradation.
I say that the compression “constantly” degrades, but it really does stop eventually. With JPEG encoding, the first save at a given quality level (e.g., save as 80%) causes the most data loss. Subsequent decode and re-encode cycles at the same quality level will result in less and less loss. The first save causes the most loss. The second causes some loss, but not as much as the first time. The third save causes less loss than the second, etc. You would probably have to resave a JPEG over a dozen times to see it normalize, but it should eventually stop degrading, unless you use Photoshop. With Adobe products, JPEGs may take thousands of resaves to normalize, and they will look very distorted.
The impact from this lossy compression is detectable. For this example, I’ll use a photo that I took yesterday…
|Resaved with Photoshop CS5 at “high” quality (first resave).|
|Resaved first resave with with Photoshop CS5 at “high” quality (second resave).
With this picture, the second resave is only a little different from the first resave. However, the amount of change between the first and second resaves really depends on the picture. The only consistency is that the second resave will not change more than the first resave.
Because nothing else was altered between saves, the first and second resaves are very similar. The first save removed most of the artifacts and the second save removed a few artifacts. (If you look in the ELA map at the cup’s lid, you may notice that some of the small white squares are gone in the second resave.)
While the picture’s content may not be very exciting, it does have a couple of great attributes:
- There are large areas of mostly white and mostly black. Solid colors compress very efficiently. As a result, the white on the lid, white sunlight on the floor, and part of the black border on the laptop’s monitor all appear solid black under ELA. These areas were so efficiently compressed in the original image that they didn’t change between resaves.
- There are visible high-contrast edges. For example, the white cup against the brown table, black laptop against the brown table, and the ribs in the dark brown chairs against the light brown wall. All of these edges have similar ELA intensities.
- There are lots of mostly flat surfaces. The white cup, the lid, most of the black laptop, the wall in the background, and even the low-contrast table (where the sunlight is not bringing out details). These are all surfaces and they are all at the same ELA intensity.
- There are textured surfaces, denoted as small regions with high-detail patterns: the text on the cup, the computer screen, the keyboard letters are visually similar (white/black) and have similar intensities.
With ELA, you want to compare similar attributes with similar attributes. Each of these areas (surfaces, edges, and textures) may compress at different rates. But in general, all similar surfaces should compress at the same rate. Edges should compress at the same rate as similar edges, and textures should compress at the same rate as similar textures.
When a picture is edited, the modified areas are likely at a different compression level than the rest of the picture. This is how we know that the sidewalk picture (beginning of this post) was digitally altered. We do not make the determination by saying “white means edited”. Instead, we identify that a section of each sidewalk cover is inconsistent with the rest of the picture. This inconsistency permits identifying the edit.
The thing to remember is that ELA maps out the error level potential — the amount of expected loss during a resave. If a picture is resaved too many times, then the compression level becomes normalized. At that point, subsequent resaves at the same quality level will not alter the picture. This results in a black, or near black ELA map.
|Original resaved at a low quality|
Unfortunately, it is still common to see people who don’t read the tutorials and claim that ELA does not work by uploading a low quality picture as their proof. Alternately, they upload a picture that has undergone global modifications (e.g., scaling or recoloring) that changes all pixel values, resulting in a higher/whiter ELA compression map. But even in these cases, ELA still functions properly — it still generates a topology map that represents the potential compression rate across the picture. This may not be useful for identifying if a picture was spliced, but it is useful for detecting what happened during the last save.
A few days ago, a group called “Bellingcat” published a report where they tried to do some digital photo forensics. They were trying to show that some satellite photos were digitally altered. They used FotoForensics to evaluate the picture, but unfortunately ended up misinterpreting the results.
In Bellingcat’s analysis, they claims that the picture was altered because the five regions (A-E) look different. However, they failed to remember to compare similar attributes:
- Region “A” shows clouds and is uniformly white. Solid colors compress really well, so the ELA result is solid black. This indicates that the uniformly colored region is already optimally compressed.
- Region “E” has a little noise surrounded by black in the ELA — just like the lid in the coffee cup example. This is where the colors blend from solid white to near white.
- Region “C” has a consistent texture. It shows land and buildings.
- Region “D” has a different texture from C. It is a smoother surface. Clouds with no texture are relatively smooth and compress better than complex textures. This results in the expected lower error level potential. This area also appears consistent with the lower-left region of “C”, where the clouds partially cover the land.
- Region “B” has… well, I see no difference between B and D.
The one thing that ELA really pulled out are the annotations. They are at a much higher error level potential, indicating that they have not been resaved as many times as the rest of the picture.
Using ELA, we cannot determine the authenticity of this picture: we cannot tell if it is real, and we cannot tell if it is fake. We can only conclude that this is a low quality picture and that the black text on white annotations were added last. If there was a higher quality version of this picture (without the annotations), then we would have a better chance at detecting any potential alterations.
Everyone’s a critic
A number of people have pointed out flaws in the Bellingcat analysis. A forensic examiner in Australia used different tools and methods than me and found other inconsistencies in the Bellingcat findings. I think Myghty has one of the most thorough debunkings of the Bellingcat report.
Unfortunately, other forensic experts chose to blame the tool rather than the uneducated users (yes Bellingcat, I’m calling you uneducated). For example, Spiegel quoted German image forensics expert Jens Kriese as saying:
From the perspective of forensics, the Bellingcat approach is not very robust. The core of what they are doing is based on so-called Error Level Analysis (ELA). The method is subjective and not based entirely on science. This is why there is not a single scientific paper that addresses it.
The ignorance spouted by Kriese offends me. In particular:
- Kriese is correct that the results from the ELA system at FotoForensics is subjective — it is up to the analyst to draw a conclusion from the generated compression map. However, this is no different than requiring a human to look through a microscope to identify cancer in a tissue sample. The scientific method is both objective and subjective. Tools should be repeatable and predictable — that is objective. ELA generates a consistent, repeatable, and predictable map of JPEG’s lossy compression potential.
In order to interpret results, we use two types of reasoning: inductive and deductive. Deductive is objective, while inductive is subjective. Inductive reasoning is often used for predicting, forecasting, and behavioral analysis. (“Did someone alter this picture?” or “did a camera generate this?” are behaviors.)
As an example, if you have ever broken a bone then you likely had an X-ray. The X-ray permits an analyst to view details that would otherwise go unseen. The X-ray is objective, not subjective. However, when the X-ray technician says, “I cannot tell you that it is broken because a diagnosis requires a doctor”, then you enter the realm of subjective. (This is why you can ask for a “second opinion” — opinions are subjective.) Similarly, ELA acts like an X-ray, permitting unseen attributes to become visible. However the interpretation of the ELA results is not automated and requires a human to make a subjective determination based on specific factors.
- Identifying artifacts is part of the scientific process. In fact, it’s the first step: observation. Given that ELA works consistently and predictably, it can also be used to test a hypothesis. Specific tests include: Do similar edges have similar ELA intensities? Do similar surfaces appear similar? And do similar textures appear similar? If the hypothesis is that the picture was altered and the ELA generates consistent error level potentials, then it fails to confirm the hypothesis. An alternative is to hypothesize that the picture is real and see an inconsistent ELA image. Inconsistency would prove the hypothesis is false, enabling an analyst to detect alterations.
For Kriese to question whether ELA is based on science, or to criticize the subjective portion of the evaluation, makes me question his understanding of the scientific method.
- Kriese says that “there is not a single scientific paper” covering ELA. Clearly Kriese has not read my blog. Four years ago I wrote about a Chinese researcher who plagiarized my work and had it published in a scientific journal: Lecture Notes in Computer Science, 2011, Volume 6526/2011, 1-11.
ELA is also mentioned in the “Digital Photo Forensics” section of the Handbook of Digital Imaging (John Wiley & Sons, Ltd). I wrote this encyclopedia’s section and it was technically reviewed prior to acceptance.
In fact, ELA was first introduced in a white paper that was presented at the Black Hat Briefings computer security conference in 2007. Since computer forensics is part of computer security, this technology was presented to peers.
That makes three scientific papers that cover ELA. I can only assume that Kriese did not both looking anything up before making this false claim.
- The entire argument, that research is not scientific unless it is published in a scientific paper, is fundamentally flawed. I have multiple blog entries about various problems with the academic publication process. Journal publication is not timely, authors often leave out critical information necessary to recreate or verify results, papers typically lack readability, trivial alterations are considered novel, and papers frequently discuss positives and omit limitations.
There are also significant flaws with the peer review process. Peer reviews often dismiss new discoveries when they conflict with the peer’s personal interests. And if peer review actually worked, then why are plagiarism, false reporting, retractions, and even fake peer reviews so prevalent?
In addition, many companies have proprietary technologies that have not been publicly published. This does not mean that the technologies are unscientific. It only means that the details are not public. (In the case of ELA, the details are public.)
It is extremely myopic for Kriese to (1) believe that something is only scientific if it is published, and (2) attribute more creditability to published science articles than they deserve.
Similarly, Hany Farid repeated his misunderstanding by saying:
The reliance on error level analysis is fatally flawed as this technique is riddled with problems that mis-characterize authentic images as altered and failed to detect alterations.
As I have repeatedly stated, the automated portion of ELA does not “detect” anything. Detection means drawing a conclusion. ELA highlights artifacts in the image, explicitly quantifies the JPEG error level potential across the image, and does it in a provable, repeatable, predictable way. The resulting compression map generated by ELA is deterministic, idempotent, and independent of personal opinion.
I also find it a little ironic that Farid’s statements, “mis-characterize authentic images as altered” and “failed to detect alterations”, can be explicitly applied to his own “izitru” and “FourMatch” commercial products. Unless the picture is a camera original, izitru will report that it could be altered. In effect, virtually everything online could be altered.
Both Kriese and Farid are correct that the Bellingcat report is bogus. However, they both incorrectly blame the problem on ELA. It’s not the tool that is in error, it’s the authors of the Bellingcat report.
See one, do one, teach one
I do not believe it is possible to teach everyone. Some people have no incentive to learn, while others have ingrained beliefs that are personally biased or based on false premises. However, this does not mean that I will stop trying to help those who want to learn.
Occasionally I debunk algorithms published in scientific journals. In the near future, I’ll cover a widely deployed forensic algorithm — that was published in a peer-reviewed journal. This algorithm is used by many forensic analysts and even taught in a few classes. But is so unreliable that it has virtually no practical value.
Interesting research: “We Can Track You If You Take the Metro: Tracking Metro Riders Using Accelerometers on Smartphones“:
Abstract: Motion sensors (e.g., accelerometers) on smartphones have been demonstrated to be a powerful side channel for attackers to spy on users’ inputs on touchscreen. In this paper, we reveal another motion accelerometer-based attack which is particularly serious: when a person takes the metro, a malicious application on her smartphone can easily use accelerator readings to trace her. We first propose a basic attack that can automatically extract metro-related data from a large amount of mixed accelerator readings, and then use an ensemble interval classier built from supervised learning to infer the riding intervals of the user. While this attack is very effective, the supervised learning part requires the attacker to collect labeled training data for each station interval, which is a significant amount of effort. To improve the efficiency of our attack, we further propose a semi-supervised learning approach, which only requires the attacker to collect labeled data for a very small number of station intervals with obvious characteristics. We conduct real experiments on a metro line in a major city. The results show that the inferring accuracy could reach 89% and 92% if the user takes the metro for 4 and 6 stations, respectively.
The Internet of Things is the Internet of sensors. I’m sure all kinds of surveillance is possible from all kinds of sensing inputs.
If you’ve been paying attention in recent years, you might have noticed that just about everyone is losing your personal data. Even if you haven’t noticed (or maybe you just haven’t actually received a breach notice), I’m here to tell you that if you’re an American, your basic personal data is already for sale. What follows is a primer on what you can do to avoid becoming a victim of identity theft as a result of all this data (s)pillage.
A seemingly never-ending stream of breaches at banks, healthcare providers, insurance companies and data brokers has created a robust market for thieves who sell identity data. Even without the help of mega breaches like the 80 million identities leaked in the Anthem compromise or last week’s news about 4 million records from the U.S. Office of Personnel Management gone missing, crooks already have access to the information needed to open new lines of credit or file phony tax refund requests in your name.
If your response to this breachapalooza is to do what each the breached organizations suggest — to take them up on one or two years’ worth of free credit monitoring services — you might sleep better at night but you will probably not be any more protected against crooks stealing your identity. As I discussed at length in this primer, credit monitoring services aren’t really built to prevent ID theft. The most you can hope for from a credit monitoring service is that they give you a heads up when ID theft does happen, and then help you through the often labyrinthine process of getting the credit bureaus and/or creditors to remove the fraudulent activity and to fix your credit score.
In short, if you have already been victimized by identity theft (fraud involving existing credit or debit cards is not identity theft), it might be worth paying for these credit monitoring and repair services (although more than likely, you are already eligible for free coverage thanks to a recent breach at any one of dozens of companies that have lost your information over the past year). Otherwise, I’d strongly advise you to consider freezing your credit file at the major credit bureaus.
There is shockingly little public knowledge or education about the benefits of a security freeze, also known as a “credit freeze.” I routinely do public speaking engagements in front of bankers and other experts in the financial industry, and I’m amazed at how often I hear from people in this community who are puzzled to learn that there is even such a thing as a security freeze (to be fair, most of these people are in the business of opening new lines of credit, not blocking such activity).
Also, there is a great deal of misinformation and/or bad information about security freezes available online. As such, I thought it best to approach this subject in the form of a Q&A, which is the most direct method I know how to impart knowledge about a subject in way that is easy for readers to digest.
Q: What is a security freeze?
A: A security freeze essentially blocks any potential creditors from being able to view or “pull” your credit file, unless you affirmatively unfreeze or thaw your file beforehand. With a freeze in place on your credit file, ID thieves can apply for credit in your name all they want, but they will not succeed in getting new lines of credit in your name because few if any creditors will extend that credit without first being able to gauge how risky it is to loan to you (i.e., view your credit file). And because each credit inquiry caused by a creditor has the potential to lower your credit score, the freeze also helps protect your score, which is what most lenders use to decide whether to grant you credit when you truly do want it and apply for it.
Q: What’s involved in freezing my credit file?
A: Freezing your credit involves notifying each of the major credit bureaus that you wish to place a freeze on your credit file. This can usually be done online, but in a few cases you may need to contact one or more credit bureaus by phone or in writing. Once you complete the application process, each bureau will provide a unique personal identification number (PIN) that you can use to unfreeze or “thaw” your credit file in the event that you need to apply for new lines of credit sometime in the future. Depending on your state of residence and your circumstances, you may also have to pay a small fee to place a freeze at each bureau. There are four consumer credit bureaus, including Equifax, Experian, Innovis and Trans Union.
Q: How much is the fee, and how can I know whether I have to pay it?
A: The fee ranges from $0 to $15 per bureau, meaning that it can cost upwards of $60 to place a freeze at all four credit bureaus (recommended). However, in most states, consumers can freeze their credit file for free at each of the major credit bureaus if they also supply a copy of a police report and in some cases an affidavit stating that the filer believes he/she is or is likely to be the victim of identity theft. In many states, that police report can be filed and obtained online. The fee covers a freeze as long as the consumer keeps it in place. Equifax has a decent breakdown of the state laws and freeze fees/requirements.
Q: What’s involved in unfreezing my file?
A: The easiest way to unfreeze your file for the purposes of gaining new credit is to spend a few minutes on the phone with the company from which you hope to gain the line of credit (or perhaps research the matter online) to see which credit bureau they rely upon for credit checks. It will most likely be one of the major bureaus. Once you know which bureau the creditor uses, contact that bureau either via phone or online and supply the PIN they gave you when you froze your credit file with them. The thawing process should not take more than 24 hours.
Q: I’ve heard about something called a fraud alert. What’s the difference between a security freeze and a fraud alert on my credit file?
A: With a fraud alert on your credit file, lenders or service providers should not grant credit in your name without first contacting you to obtain your approval — by phone or whatever other method you specify when you apply for the fraud alert. To place a fraud alert, merely contact one of the credit bureaus via phone or online, fill out a short form, and answer a handful of multiple-choice, out-of-wallet questions about your credit history. Assuming the application goes through, the bureau you filed the alert with must by law share that alert with the other bureaus.
Consumers also can get an extended fraud alert, which remains on your credit report for seven years. Like the free freeze, an extended fraud alert requires a police report or other official record showing that you’ve been the victim of identity theft.
An active duty alert is another alert available if you are on active military duty. The active duty alert is similar to an initial fraud alert except that it lasts 12 months and your name is removed from pre-approved firm offers of credit or insurance (prescreening) for 2 years.
Q: Why would I pay for a security freeze when a fraud alert is free?
A: Fraud alerts only last for 90 days, although you can renew them as often as you like. More importantly, while lenders and service providers are supposed to seek and obtain your approval before granting credit in your name if you have a fraud alert on your file, they’re not legally required to do this.
Q: Hang on: If I thaw my credit file after freezing it so that I can apply for new lines of credit, won’t I have to pay to refreeze my file at the credit bureau where I thawed it?
A: Yes (unless you’ve previously qualified for a free freeze). However, even if you have to do this once or twice a year, the cost of doing so is almost certainly less than paying for a year’s worth of credit monitoring services.
Q: Is there anything I should do in addition to placing a freeze that would help me get the upper hand on ID thieves?
A: Yes: Periodically order a free copy of your credit report. By law, each of the three major credit reporting bureaus must provide a free copy of your credit report each year — via a government-mandated site: annualcreditreport.com. The best way to take advantage of this right is to make a notation in your calendar to request a copy of your report every 120 days, to review the report and to report any inaccuracies or questionable entries when and if you spot them.
Q: I’ve heard that tax refund fraud is a big deal now. Would having a fraud alert or security freeze prevent thieves from filing phony tax refund requests in my name with the states and with the Internal Revenue Service?
A: Neither would stop thieves from fraudulently requesting a refund in your name. However, a freeze on your credit file would have prevented thieves from using the IRS’s own Web site to request a copy of your previous year’s tax transcript — a problem the IRS said led to tax fraud on 100,000 Americans this year and that prompted the agency to suspend online access to the information. For more information on what you can do to minimize your exposure to tax refund fraud, see this primer.
Q: Okay, I’ve got a security freeze on my file, what else should I do?
A: It’s also a good idea to notify a company called ChexSystems to keep an eye out for fraud committed in your name. Thousands of banks rely on ChexSystems to verify customers that are requesting new checking and savings accounts, and ChexSystems lets consumers place a security alert on their credit data to make it more difficult for ID thieves to fraudulently obtain checking and savings accounts. For more information on doing that with ChexSystems, see this link.
Q: If I freeze my file, won’t I have trouble getting new credit going forward?
A: If you’re in the habit of applying for a new credit card each time you see a 10 percent discount for shopping in a department store, a security freeze may cure you of that impulse. Other than that, as long as you already have existing lines of credit (credit cards, loans, etc) the credit bureaus should be able to continue to monitor and evaluate your creditworthiness should you decide at some point to take out a new loan or apply for a new line of credit.
Q: Anything else?
A: ID thieves like to intercept offers of new credit and insurance sent via postal mail, so it’s a good idea to opt out of pre-approved credit offers. If you decide that you don’t want to receive prescreened offers of credit and insurance, you have two choices: You can opt out of receiving them for five years or opt out of receiving them permanently.
To opt out for five years: Call toll-free 1-888-5-OPT-OUT (1-888-567-8688) or visit www.optoutprescreen.com. The phone number and website are operated by the major consumer reporting companies.
To opt out permanently: You can begin the permanent Opt-Out process online at www.optoutprescreen.com. To complete your request, you must return the signed Permanent Opt-Out Election form, which will be provided after you initiate your online request.
A couple of years back, I was signed up for a credit monitoring service and had several unauthorized applications for credit filed in my name in rapid succession. Over a period of weeks, I fielded numerous calls from the credit monitoring firm, and spent many grueling hours on the phone with the firm’s technicians and with the banks that had been tricked into granting the credit — all in a bid to convince the latter that I had not in fact asked them for a new credit line.
The banks in question insisted that I verify my identity by giving them all of my personal information that they didn’t already have, and I was indignant that they should have been that careful before opening the new fraudulent accounts. Needless to say, the experience was extremely frustrating and massively time-consuming.
We eventually got that straightened out, but it took weeks. Not long after that episode, I decided to freeze my credit and that of my wife’s at all of the major bureaus. Turns out, I did that none too soon: A few weeks later, I broke a story about a credit card breach at nationwide beauty chain Sally Beauty, detailing how the cards stolen from Sally Beauty customers had wound up for sale on Rescator[dot]cc, the same fraud shop that had been principally responsible for selling cards stolen in the wake of the massive data breaches at Home Depot and Target.
In response to my reporting about him and his site, Rescator changed his site’s home page to a photoshopped picture of my driver’s license, and linked his customers (mostly identity thieves and credit card hustlers) to a full copy of my credit report along with links to dozens of sites where one can apply for instant credit. Rescator also encouraged his friends and customers to apply for new credit in my name.
Over the next few weeks, I received multiple rejection letters from various financial firms, stating that although they had hoped to be able to grant my application for new credit, they were unable to do so because they could not view my credit file. The freeze had done its job.
In summary, credit monitoring services are helpful in digging you out of an identity theft ditch. But if you want true piece of mind, freeze your credit file.
The Birth of DHT, May 2005
When BitTorrent started in 2002, decentralization was one of its main innovations. The central structure of services like Napster ultimately led to their downfall, and while decentralized systems such as eDonkey/eMule and Gnutella existed, they were often cumbersome and filled with fakes and spam.
BitTorrent was also somewhat individualized. Clients only dealt with clients on the swarms they were interested in, and all conducted business through a tracker.
This led to problems though, when trackers went down, as the trackers were the only way for peers to get information about others in the swarm. There was no fallback, except trying to add more trackers and hope everyone else adds the same. However, with the launch of Distributed Hash Tables (DHT) these problems were all but over.
That two similar but incompatible DHT systems were launched within weeks of each other is quite surprising, given the history behind both. To this day, in fact, the systems are still incompatible, although there are plug-ins that allow the use of both to act as a bridge between the two swarms (one Vuze, one Mainline).
When you factor in that both were released just months after eXeem had tried and failed to do a similar thing (earning significant criticism while doing so) the success and longevity of both look even more impressive. But how did they come about?
The Vuze DHT debuted first, with version 2.3.0 of the Azureus client on May 2, 2005. In its announcements back then, they were keen to stress the difference from eXeem, stating it was a decentralized layer on top of BitTorrent, rather than a decentralised BitTorrent system itself. Within 24 hours there were more than 200,000 peers, and there are currently around 1.1 million peers on the network.
According to Paul Gardener, the main developer of the Azureus DHT system, tracker redundancy wasn’t the main reason behind its development. Instead, decentralization for search was driving it.
“That was one of my pet aims when I joined the Azureus development team,” Gardener told TF earlier this month. “But the others in the team weren’t sure if search was a priority, so I found a way of working on some decentralization that perhaps one day could evolve into/be adapted for search. Of course decentralized tracking was a good aim in itself.”
“I started from scratch,” Gardener recalls, “there weren’t any libraries out there I could use, so had to figure out which kind of DHT to use (Kademlia) etc. [It took] a few months I guess.”
Three weeks later, Bittorrent Inc. released their own version of DHT with the release of version 4.1. This was then adopted by the then popular client BitComet in early June, and by other clients soon after.
While the timing may suggest otherwise, BitTorrent’s DHT wasn’t a response to Vuze’s release at all, as the person responsible – Drue Loewenstern – had been working on it since 2002.
“I started working on the DHT in the summer of 2002 after making the first Mac BitTorrent clients, a year before Azureus was established on Sourceforge. Finishing it off and integration into BitTorrent started in 05 when BT became a company. I was in testing and about to release it when Azureus launched theirs,” Loewenstern says.
The inspiration for the BitTorrent mainline DHT came from an unlikely and famous source: Aaron Swartz.
“Distributed hash tables were an inspiring area of research. I was really into P2P, having just worked on MojoNation and BitTorrent, and wanted to do all sorts of cool decentralized things like trust metrics. Aaron Swartz, 15 at the time, circulated a one page implementation of the Chord algorithm and I was struck by its simplicity, Loewenstern notes.
“I started looking into DHTs specifically and Kademlia was the first DHT paper that really clicked with me and seemed like it might work in the real world So I decided to start implementing it without really knowing what I was going to do with it.”
Contrary to Vuze, redundancy was one of the main motivations driving the development of the mainline DHT.
In the case of BitTorrent, the goal of the DHT has always been to make BT more robust, to improve performance by finding more peers, and to simplify publishing by making a tracker optional,” Loewenstern says.
Of course, not everyone was thrilled to see the introduction of DHT. Private trackers were opposed to DHT as it enabled people to use the site’s torrents without being under the strict control of the tracker admins.
The solution to this was a form of access control called the private flag, which disabled DHT, along with Peer Exchange (PEX) and restricted peer access to trackers – locking things into the way of 2005.
The flag works by being inside the data used to generate the hash, so if disabled, it would change the overall torrent hash, meaning a torrent with the flag enabled would be a completely separate swarm from one with the flag disabled. It also gave these sites a new way to market themselves, by taking the term “private flagged torrent trackers” and condensing it to “private trackers,” implying some form of privacy.
This move though, was not by choice.
“There’s always been tensions between clients and private trackers,” Vuze’s Gardener says. “In particular they like to ban Vuze because it is ‘open source and people have hacked it to report incorrect stats’ or other such ‘reasons’. I’ve never been a fan of [the private flag] as a solution.”
“It came to be because some index site operators enforce upload/download ratios in an effort to keep seeders around for torrents that nobody wants to be left holding the bag for by seeding. They thought DHT (and PEX) might let users bypass the ratio system so they made a lot of noise about banning clients that implemented DHT,” he says.
“Azureus didn’t want to get banned so they came up with the private flag and added it to their client. It wasn’t my decision to add it to BitTorrent. Without PEX, torrents take longer to ramp up so it annoys me when people upload private torrents to public index sites.”
NEXT, The BitComet Incident
As the spiritual home of The Pirate Bay and the birthplace of some of the world’s most hardcore file-sharers, Sweden has definitely earned its place in the history books. If Swedes can be converted to legal offerings, just about anyone can, one might argue.
A new study just published by the Film and TV Industry Cooperation Committee (FTVS) in collaboration with research company Novus reveals some interesting trends on local media consumption habits.
Covering both legal and illegal services, the survey is based on 1,003 interviews carried out between Feb 27 and March 9 2015 among citizens aged 16 to 79-years-old.
Legal and illegal consumption
On the legal TV and movie consumption front, Sweden appears to be doing well. A decent 71% of respondents said they buy services such as Netflix and HBO, with a quarter using such services every day and 35% watching several times each week.
In comparison, 29% of all respondents admitted to using illegal services to watch film and television. Perhaps unsurprisingly the activity is most prevalent among the young, with 60% of 16 to 29-year-olds confessing to using pirate sites.
The survey found that around 280 million movies and TV shows are watched illegally in Sweden each year, with respondents indicating they would have paid for around a third of those if illegal services weren’t available.
With torrents extremely popular around Europe, it’s interesting to note that downloading of content is now taking second place to online streaming. The survey found that 19% of respondents stream content illegally, while 17% download. When users engage in both streaming and downloading, streaming is the more popular activity.
The study notes that dual users (those that use both legal and illegal services) watch every third movie or TV show illegally, an average of four films and seven TV shows every month.
The survey also polled respondents on their attitudes to piracy. Six out of ten respondents said they think that using ‘pirate’ sites to watch movies and TV shows is “wrong”. Four out of ten agreed, but previously used these services anyway.
On the thorny question of what to do about piracy, respondents were asked what they thought would be the best solution.
Somewhat conveniently for an anti-piracy focused report, 43% of respondents indicated that ISPs should play a part in reducing the numbers of user visiting illegal services, with 24% opting for site blocking measures and 19% suggesting a warning notice scheme.
However, when it comes to the heavy hand of the law, a minority of respondents show an interest. Just 10% believe that boosting law enforcement and judicial resources will solve the problem while a tiny 4% think that harsher punishments will bring results.
Commenting on the report, Per Strömbäck of FTVS says that the situation in Sweden is far from satisfactory.
“There is a common misconception that piracy is less of a problem today because we have a wide range of legal options. On the contrary, the problem of illegal services is greater than ever,” Strömbäck says.
“The situation is not sustainable. For us to be able to continue to produce, distribute and show films and TV audiences want to see and pay for, we need a functioning digital market and measures to stop the illegal competition.”
With site blocking firmly on the agenda in Sweden, entertainment industry groups will be pinning their hopes on success in the courts since there is clearly no appetite for punishing the public.
The Pirate Bay is the most censored website on the Internet. Countries all around the world have ordered Internet providers to block subscriber access to the torrent site, with Russia being the latest addition.
The idea behind these blockades is that they will help to decrease online piracy. However, a new study published by Carnegie Mellon University and Wellesley College researchers, suggests that blocking one site isn’t very effective.
The researchers used data collected by an anonymous Internet consumer panel tracking company to compare the browsing habits of UK citizens, both before and after The Pirate Bay was blocked by major ISPs in 2012.
After comparing the results to a control group and ruling out various other variables, the researchers conclude that there is no significant effect on legal consumption.
Instead, Pirate Bay users chose to circumvent the measures by using VPNs, proxies, or switching to other pirate sites.
“Our results show that blocking The Pirate Bay had little impact on consumption through legal channels — instead, consumers seemed to turn to other piracy sites, Pirate Bay ‘mirror’ sites, or Virtual Private Networks that allowed them to circumvent the block.”
While the above findings support the many opponents of website blocking, it’s only part of the story. The researchers also analysed data after a subsequent blockade that covered more than a dozen large pirate sites at once.
The results here were quite different, with a significant uptick in the number of visits (of ‘pirates’) to legal movie services such as Netflix.
“…blocking 19 different major piracy sites caused users of those sites to increase their usage of paid legal streaming sites such as Netflix by 12% on average,” the researchers write.
This effect was most pronounced for people who used the pirate sites most frequently. According to the researchers this makes sense as they were most affected by the blockade.
“The lightest users of the blocked sites increased their clicks on paid streaming sites by 3.5% while the heaviest users of the blocked sites increased their paid streaming clicks by 23.6%, strengthening the causal interpretation of the results.”
Overall the results show that blocking The Pirate Bay in isolation is futile. For website blockades to have a serious impact they should be directed at a broad selection of pirate sites, making it harder for people to find illegal alternatives.
“Our results suggest that website blocking requires persistent blocking of a number of piracy sites in order to effectively migrate pirates to legal channels,” the researchers note.
Perhaps just as importantly, the researchers add that copyright holders should also make legal content more attractive in order to convert pirates into paying customers.
It has to be noted that the research was carried out as part of Carnegie Mellon University’s Initiative for Digital Entertainment Analytics (IDEA), which received a generous donation from the MPAA. However, the researchers suggest that their work is carried out independently.
The results may not help efforts to demand isolated Pirate Bay blockades, which is common in most countries. However, they can be used as ammunition to demand wider website blockades, which is arguably even better from a copyright holder perspective.