Our new theory of change

Post Syndicated from Ben Durbin original https://www.raspberrypi.org/blog/theory-of-change-2024/

One of the Raspberry Pi Foundation’s core values is our focus on impact. This means that we are committed to learning from the best available evidence, and to being rigorous and transparent about the difference we’re making.

A smiling girl holding a robot buggy in her lap

Like many charities, an important part of our approach to achieving and measuring our impact is our theory of change. We are excited to launch a newly refreshed theory of change that reflects our mission and strategy to ensure that young people can realise their full potential through the power of computing and digital technologies.

What is a theory of change?

A theory of change describes the difference an organisation aims to make in the world, the actions it takes to achieve this, and the underlying assumptions about how its actions will create change.

Two learners sharing a laptop in a coding session.

It’s like a good cake recipe. It describes the ingredients and tools that are required, how these are combined, and what the results should be. But a theory of change goes further: it also addresses why you need the cake in the first place, and the reasons why the recipe will produce such a good cake if you follow it correctly!

What is the change we want to make?

Our theory of change begins with a statement of the problem that needs solving: too many young people are missing out on the enormous opportunities from digital technologies, and access to opportunities to learn depends too much on who you are and where you were born.

We want to see a world where young people can take advantage of the opportunities that computers and digital technologies offer to transform their own lives and communities, to contribute to society, and to help address the world’s challenges.

Learners in a computing classroom.

To help us empower young people to do this, we have identified three broad sets of outcomes that we should target, measure, and hold ourselves accountable for. These map roughly to the COM-B model of behaviour change. This model suggests that for change to be achieved, people need a combination of capabilities, opportunities, and motivation.

Our identified outcomes are that we support young people to:

  1. Build knowledge and skills in computing
  2. Understand the opportunities and risks associated with new technologies
  3. Develop the mindsets to confidently engage with technological change

How do we make a difference?

We work at multiple levels throughout education systems and society, which together will achieve deep and long-lasting change for young people. We design learning experiences and initiatives that are fun and engaging, including hundreds of free coding and computing projects, the Coolest Projects showcase for young tech creators, and the European Astro Pi Challenge, which gives young people the chance to run their computer programs in space.

Three learners working at laptops.

We also support teachers, youth workers, volunteers, and parents to develop their skills and knowledge, and equip them to inspire young people and help them learn. For example, The Computing Curriculum provides a complete bank of free lesson plans and other resources, and Experience AI is our educational programme that includes everything teachers need to deliver lessons on artificial intelligence and machine learning in secondary schools.

Finally, we aim to elevate the state of computing education globally by advocating for policy and systems change, and undertaking our own original research to deepen our understanding of how young people learn about computing.

How will we use our theory of change?

Our theory of change is an important part of our approach to evaluating the impact of our resources and programmes, and it informs all our monitoring and evaluation plans. These plans identify the questions we want to answer, key metrics to monitor, and the data sources we use to understand the impact we’re having and to gather feedback to improve our impact in future.

An educator teaches students to create with technology.

The theory of change also informs a shared outcomes framework that we are applying consistently across all of our products. This framework supports planning and helps keep us focused as we consider new opportunities to further our mission.

A final role our theory of change plays is to help communicate our mission to other stakeholders, and explain how we can work with our partners and communities to achieve change.

You can read our new theory of change here and if you have any questions or feedback on it, please do get in touch.

The post Our new theory of change appeared first on Raspberry Pi Foundation.

Hardware Vulnerability in Apple’s M-Series Chips

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/03/hardware-vulnerability-in-apples-m-series-chips.html

It’s yet another hardware side-channel attack:

The threat resides in the chips’ data memory-dependent prefetcher, a hardware optimization that predicts the memory addresses of data that running code is likely to access in the near future. By loading the contents into the CPU cache before it’s actually needed, the DMP, as the feature is abbreviated, reduces latency between the main memory and the CPU, a common bottleneck in modern computing. DMPs are a relatively new phenomenon found only in M-series chips and Intel’s 13th-generation Raptor Lake microarchitecture, although older forms of prefetchers have been common for years.

[…]

The breakthrough of the new research is that it exposes a previously overlooked behavior of DMPs in Apple silicon: Sometimes they confuse memory content, such as key material, with the pointer value that is used to load other data. As a result, the DMP often reads the data and attempts to treat it as an address to perform memory access. This “dereferencing” of “pointers”—meaning the reading of data and leaking it through a side channel—­is a flagrant violation of the constant-time paradigm.

[…]

The attack, which the researchers have named GoFetch, uses an application that doesn’t require root access, only the same user privileges needed by most third-party applications installed on a macOS system. M-series chips are divided into what are known as clusters. The M1, for example, has two clusters: one containing four efficiency cores and the other four performance cores. As long as the GoFetch app and the targeted cryptography app are running on the same performance cluster—­even when on separate cores within that cluster­—GoFetch can mine enough secrets to leak a secret key.

The attack works against both classical encryption algorithms and a newer generation of encryption that has been hardened to withstand anticipated attacks from quantum computers. The GoFetch app requires less than an hour to extract a 2048-bit RSA key and a little over two hours to extract a 2048-bit Diffie-Hellman key. The attack takes 54 minutes to extract the material required to assemble a Kyber-512 key and about 10 hours for a Dilithium-2 key, not counting offline time needed to process the raw data.

The GoFetch app connects to the targeted app and feeds it inputs that it signs or decrypts. As its doing this, it extracts the app secret key that it uses to perform these cryptographic operations. This mechanism means the targeted app need not perform any cryptographic operations on its own during the collection period.

Note that exploiting the vulnerability requires running a malicious app on the target computer. So it could be worse. On the other hand, like many of these hardware side-channel attacks, it’s not possible to patch.

Slashdot thread.

Прозрачността в градската среда и това да си деен не само по избори

Post Syndicated from Боян Юруков original https://yurukov.net/blog/2024/sofia-deinost/

Все се говори за прозрачност и отчетност, но донякъде иронично остават доста мъгляви понятия. Писал съм за това доста през годините, например за фонда за лечение на деца, връзките на лекари с фарма компании и размислите на Радев дали е нужна прозрачност въобще. Тук опивам три малки примера какво се научава и какво можеш да направиш, ако имаш достъп, достъпност и поставени в контекст данни.

Това са все случаи с места в София, покрай които минавам редовно и затова съм се решил да се занимавам. Накрая ще добавя списък с ресурси, които може да ви помогнат да научите повече за средата около себе си. За съжаление, за София конкретно, тъй като във всички останали градове почти липсва информация.

В първия случай става въпрос за общински имот, на който има стар блок. Зад блока има паркинг. Преди няколко години живущите там са решили да преградят достъпа до имота усвоявайки го и създавайки частен паркинг. Слагат и автоматична врата с контролиран достъп. Преди 5 години някой подава сигнал, констатира се нарушение, установява се точно кой го е извършил, но заповедта се прехвърля между район Изгрев и дирекции в общината докато накрая потъва. Сега имат нов сигнал за същото и отговорът от район Изгрев е общо взето „тогава строителен контрол каза, че няма нарушение и оставихме случая“. Обаче, ако някой намери стария сигнал и прочете какво всъщност е написано, открива, че район Изгрев подвеждат и наистина са седели върху установен и документиран от институциите проблем с общинска земя години наред. Сега ДНСК е изискала нова проверка и действия по въпроса и ги чака.

Вторият случай пак е за общински имот. По-точно три, но ще се фокусираме върху най-големия. Този път инвеститор си е построил блок и е заградил съседните общински имоти. Известно време ги използва за строителни отпадъци, но после просто ги подготвя за строеж. На тези имоти незнайно как има одобрен градоустройствен план за жилищно строителство. Липсват документите за това на сайта на общината, но е добре известно, че става дума за общинска частна собственост. Явно е подготвено за продажба при „удобен общински съвет“. Това беше коментирано из групите на квартала и район Изгрев имат сигнал, на който отговориха вчера след двумесечно забавяне. Установили нарушението и нарушителя и му дали срок за доброволно премахване на оградата и освобождаване на имота. Не става ясно какъв е срокът и кога изтича. Малката подробност е, че аналогични сигнали от различни хора от квартала има 18 месеца по-рано. Тогава същите хора от район Изгрев по същия начин открили нарушение и е дали пак срок за същото. Отново не става ясно какъв, но година и половина по-късно не само няма предприети действия, но въпросният инвеститор си е насадил временни декоративни храстчета, които даже пораснаха, за да скрият какво се прави в имота. Има нов сигнал в общината за бездействието на района и непроследяването на собствените им актове.

Третият случай отново е за инвеститор, този път небезизвестен и с топли връзки с ключови хора в общината. В подготовка на нов техен проект, за да няма шанс някой да го оспорва, си осигуриха отделяне на парцел с широчина три метра по протежението на имота. Така се отделиха от „съседите“, които не бяха уведомени за последващите визи и разрешения, както е по закон. Класическа схема. Така си осигуриха разрешение за строителство само в големия парцел. Прекрасно – прецакаха системата. Последваха редица проблеми дори само със започването на строежа. За повечето бяха предупредени от район Изгрев и не бяха санкционирани. Нещо, което не успяха да скрият е, че са започнали строителни дейности и в малкия парцел, т.е. изкопаха основите и изляха стена на подземните гаражи видимо разширявайки ги в рамките на ония „юридически-санитарен“ имот и извън параметрите на проекта. Общината има сигнал, но единственото, което строителен контрол констатира е, че има разрешително за големия парцел. Премълчават дали се отнася до малкия и дали са направили проверка на място. Отново ДНСК ги чака за отговор. Ако ще се спазва буквата на закона, трябва да важи за всеки етап от строителството. Тук проблемът е, че ако не бъде отстранено това нарушение, на практика проектът не би следвало да получи акт 16, освен ако не прибегнат до нелегалните практики с които се свързва името им в медиите. Предвид как е предвиден проекта и какво се строи се очакват още проблеми най-вече свързани със задължителното озеленяване и дълбочина на почвен слой. Всичко това обаче се очаква да мине безпроблемно проверки също както аналогичните в други подобни проекти.

Всяко от тези неща изглежда дребно на пръв поглед, но са всъщност чести нарушения в София и други градове. Масово не се съблюдават изискванията, не се извършват реални проверки или когато е установено нарушение, не се санкционира или проследява отстраняването му от органите на общината. Понякога това заради чадър от ръководството, понякога е заради саботирани дела, но най-често е заради нехайство, липса на прозрачност и проследяване, а и откровена корупция по ниските нива. Това позволява особено на едри строители, за които иначе се предполага, че са им изрядни документите, да правят куп нарушения, включително с цел позволяване на строителство. Това включва подкупи за получаване на становище за пожарна безопасност, ток и вода, „услуги“ и скриване на процедурите по одобрение, както и уреждане на по-добри параметри за строеж и продажба на общинска земя директно от общинския съвет.

Лесно е да се каже, че в общината са некадърни и „не е виновен тоя дето яде баницата“. В действителност това е модел на поведение, който не се оправя с избори, а с ежедневни усилия, както всяко друго нещо в държавата ни. Означава и че действията на чиновници и техните ръководители също се диктува от това какво може да им се размине. Затова, ако искате да направите нещо отвъд гневните емоджита и писане по групи във фейса, ето няколко ресурса, от които да започнете без да ви отнема много време:

  • isofmap.bg – това е карта с повечето слоеве на градско планиране и застрояване, които самата община гледа. Там ще намерите информация за собственост на имоти, инфраструктура и прочие. От слоевете ще намерите и карта на планирано и съществуващо застрояване. Отчасти на данните в тази карта се базира 3D картата, която започнах
  • Регистрите на НАГ – тук има документи за всякакви фази и аспекти от строителството. Включително разрешения за поставяне и санкции.
  • Картата на НАГ – всеки документ от онзи регистър би трябвало да е на тази карта. Подобно на isofmap има доста слоеве, но като натиснете на имот и показва известни свързани документи. За съжаление, 30% от документите не са свързани с картата, но доста от новите са
  • call.sofia.bg – сигнали към общината за нередности и проблеми в градската среда.
  • Визуализацията към call.sofia – тъй като на портала на общината не се откриват стари сигнал и не е особено лесен за търсене по принцип, направих тази визуализация. С нея открих и стаите сигнали, за които говоря по-горе. Пак на база тези данни направих карта на сигналите за паднали клони в София тази зима.
  • Картата на GovAlert – тъй като регистрите и дори картата на НАГ не са особено лесни за търсене и ориентиране, направих тази карта, която се обновява в реално време. Има филтри за време и видове документи. Също се опитва да свърже документи, които не са включени в картата на НАГ като търси идентификатори на имоти. Повече ще прочетете в статията ми когато започнах проекта преди три години.
  • Twitter акаунта на @NAG_SO_public – тъй като не всички документи може да се сложат на карта, всичко се пуска допълнително и в този акаунт като новини. Добавени са също прес съобщения от НАГ, обявления за обществени обсъждания и прочие.
  • Сигурно електронно връчване – за всякаква комуникация с институции е полезно. Трябва само ПИК на НАП или електронен подпис в момента. Докато при сигналите към общината няма опция за отговор или ескалиране на проблем, тук може лесно да се препращат сигнали, добавя информация и оспорват резултати. Също да се изпраща запитвания по ЗДОИ. С този портал няма опция да не входират документ, защото всичко е проследимо и подписано електронно.

Ресурсите за градоустройство в София всъщност предоставят някаква прозрачност и данни, макар да е все още трудна за следене и преглеждане. В другите градове дори това липсва. Доколкото строителството там беше несравнимо по-малко дори спрямо населението и размерите на града, това се променя в последните години особено в Пловдив и Бургас. Затова добавих Пловдив и Благоевград и работя все още да добавя Бургас и Варна към картата и бюлетина на отделен twitter акаунт – @BGCityPlanning. Все пак, поради значително по-малкия поток от информация, може да следите протоколите и обявите на сайта на общината си докато нещо по-добро е налично.

The post Прозрачността в градската среда и това да си деен не само по избори first appeared on Блогът на Юруков.

Грешка 404: Здравеопазване по неправилно зададени параметри

Post Syndicated from Надежда Цекулова original https://www.toest.bg/greshka-404-zdraveopazvane-po-nepravilno-zadadeni-parametri/

Грешка 404: Здравеопазване по неправилно зададени параметри

Мария Шаркова е адвокат, специализиращ в сферата на медицинското право. През 2019 г. създава уебсайта „Лекарска грешка“, в който събира статистика за делата по случаите, наречени в правото „медицински деликти“. Тя обаче смята, че в крайна сметка важното е не да се регистрират т.нар. лекарски грешки, а дали може да не се стига до подобни събития. В последното си изследване адв. Шаркова се занимава с въпроса как е гарантирано качеството на медицинската помощ в българската нормативна уредба. 

Защо се занимавате с изследвания, свързани с качеството на медицинската помощ?

През 2016 г. учих в САЩ и времето, което изкарах там, ми даде да разбера, че в България не събираме никакви данни, свързани с инцидентите в лечебните заведения и въобще с качеството на медицинската помощ. А когато липсват данни, не може да се вземат аргументирани решения, всичко е само лично мнение. 

Когато се върнах, започнах да изследвам съдебната практика при делата на пациенти срещу лекари и лечебни заведения. Правя това и до ден днешен. Натрупах богата база данни от близо 600 дела. Това са хиляди постановени съдебни актове, защото по едно дело може да се постановят и три съдебни акта, ако делото мине на три инстанции. Първият етап беше да извадя ясните количествени индикатори – колко дела са се водили за определен период, кои специалности стават най-често обект на съдебни дела, каква част от тези дела са приключили в полза на пациента и т.н.

Следващият етап дойде логично с въпросите какво е качествена медицинска помощ, кога пациентът получава некачествена медицинска помощ и респективно каква е дължимата грижа, която пациентите имат право да очакват от медицинския персонал и от лечебното заведение. Това не са абстрактни въпроси, те би трябвало да имат ясен отговор в нормативната уредба и в съдебната практика. 

Установихте ли такъв ясен отговор при изследването си?

За съжаление, установих, че има много неясноти и много противоречия – както в съдебната практика, така и в правната теория, а и сред самите медицински специалисти. Има много неизяснени въпроси или недокрай изяснени, или такива, които са регламентирани противоречиво.

Наскоро публикувахте статия по този въпрос в сп. „Медицинско право и здравеопазване“ (бр. 1/2024), но читателите на „Тоест“ са с разнообразни компетентности и вероятно малцина от тях са юристи. След като Вие като юрист откривате толкова неясноти, може ли да се формулира достъпен съвет какво има право да изисква всеки от нас като пациент?

Това, което е ясно и което е написано в закона, е, че медицинската помощ в България трябва да се оказва въз основата на утвърдени от науката и практиката методи. Тоест когато пациент отиде в лечебно заведение, той има право да очаква, а медицинският персонал е длъжен да му оказва такава медицинска помощ, която е утвърдена и в медицинската теория, и в практиката. 

И тук вече идва въпросът какво означава това. Как разбираме къде е записано по какъв начин се утвърждават тези методи?

Ще Ви дам един добре познат пример. Представете си, че една жена отива да ражда. Допустимо ли е по време на раждането да ѝ приложат метода „Кристелер“, който българските родилки познават като „скачане по корема“ и е описан в учебник по акушерство и гинекология от 1967 г., или медиците, асистиращи на раждането, трябва да се съобразяват със съвременната наука, според която този метод е отречен и се смята за опасен, а в много държави е и изрично забранен? Българската уредба не ни дава ясен отговор. 

Друг проблем, който е свързан с оказването на медицинска помощ, е в случаите, в които пациентът смята, че има някакъв пропуск в неговото лечение. За да се установи дали това е така, съдът назначава вещи лица. Когато едно вещо лице преценява дали лечението е било правилно, или дали диагнозата е била правилно поставена, или дали се е процедирало както трябва, то обикновено изразява своето мнение, основано на собствения му опит. Много рядко вещите лица се опират на утвърдените от науката и практиката методи. В доста случаи те дори не посочват научните източници, на които основават твърдението си, че пациентът е получил качествена или некачествена медицинска помощ. 

Така че – да се върна към въпроса Ви – всеки от нас има право да изисква и да получи медицинска помощ съобразно утвърдените от науката и практиката методи. За съжаление, никой от нас не може със сигурност да знае кои са тези методи, защото това не е изрично разписано.

В Закона за здравето е записано, че качеството на медицинската помощ се основава на медицински стандарти и Правила за добра медицинска практика. Не следва ли именно в тези документи да пише кои са утвърдените от науката и практиката методи, за които говорите? 

По принцип следва. Но тук идва следващият проблем. В Закона за лечебните заведения пише, че тези медицински стандарти трябва да се приемат и утвърждават от министъра на здравеопазването. Но няма единен модел, по който те да бъдат разработвани, не е изяснено какво трябва да е тяхното съдържание, и не е регламентиран редът, по който се изготвят проектите на тези стандарти, докато стигнат до утвърждаване с наредба от министъра на здравеопазването. Така стигаме до съществуващата практика голяма част от стандартите да се изготвят на тъмно, без широкото участие на медицинската общност. Понякога в тях може да се открият лобистки текстове, които отразяват интересите на много тесен кръг от специалисти, участвали в изготвянето им.

Може ли да дадете пример? Трудно е да си представи човек как един медицински стандарт може да обслужва нечий интерес.

Например преди две-три години с промени във всички медицински стандарти беше премахнато изискването за брой медицински персонал, който задължително трябва да има в едно звено, било то клиника или отделение. Това беше оставено на преценката на лечебните заведения в зависимост от тяхната натовареност и необходимост с оглед конкретен пациент. Но това е едно много широко понятие, а броят на персонала много често има пряко отношение към качеството на медицинската помощ. Ако в една клиника или отделение има 30 или 35 пациенти, не е редно да се остави на преценката на лечебното заведение дали те ще бъдат обслужвани от петима специалисти, или от един специализант.

Извън тези количествени показатели не са ли разписани в медицинските стандарти конкретни „методи, утвърдени в науката и практиката“?

Най-често не са. Иначе казано, в момента пациентите в България по закон имат право да получат качествена медицинска помощ, но ако по някаква причина сметнат, че това тяхно право е нарушено, няма как да го докажат. Това ще зависи от субективната преценка на вещите лица. 

Да се върнем към примера ни за метода „Кристелер“. Ако  родилката, която споменахме по-рано в разговора, прецени, че като са ѝ приложили този метод, това не отговаря на изискването за качествена медицинска помощ, защото например ѝ е руптурирала матката или са ѝ счупили две ребра, или има други травми, тя трябва да докаже дали този метод е утвърден в науката и практиката, или не. Но в едно съдебно дело отговорът на въпроса ще зависи изцяло от вещото лице. И ако вещото лице е спряло своето професионално развитие с учебника от 1967 г., ще отговори, че да, това е един възможен метод, утвърден, и няма нищо лошо в използването му. 

Както пациентът зависи от това дали дадено вещо лице е актуализирало познанията си, по същия начин при евентуален правен спор зависи и медицинският специалист. Попадала съм на становище на вещо лице, в което пише, че в медицината всичко е възможно. Попадала съм на становище, в което лабораторните изследвания са сравнени с „невярна любовница“. 

Когато говорим за тези утвърдени от науката и практиката методи, те най-често се описват в документи, наречени протоколи или алгоритми за клинично поведение. В тях се описват действията, които трябва да се извършат в конкретна клинична ситуация, защото пациентът по принцип има право на „своевременна и достатъчна“ медицинска помощ, но за всяка конкретна ситуация кое е своевременно и кое е достатъчно, се описва именно в тези протоколи. В голяма част от медицинските стандарти обаче не се говори въобще за протоколи и алгоритми за поведение. В други се насочва към протоколите на конкретно професионално дружество, без да е ясно защо се реферира към тях. В трети се изисква всяко лечебно заведение само̀ да си изработи такива протоколи, а по някои специалности въобще няма разписани стандарти. Това създава един водовъртеж от случайни обстоятелства, в центъра на които попада пациентът.

Има ли добър пример?

Спешната медицина е единствената сфера, в която има утвърдени със заповед на министъра на здравеопазването правила за клинично поведение. Това е така, защото е предвидено и в самия стандарт. 

Разговорът ни не звучи оптимистично дотук. Излиза, че като пациенти не сме особено защитени и разчитаме изцяло на личностните качества на медицинските специалисти, при които ще попаднем.

Напомняте ми да отбележа, че медицинските специалисти също не са добре защитени. Защото и те разчитат на личностните си качества и ако при тях попадне случай, който надхвърля компетенциите им, те са в риск, защото няма начертана система, която да ги подкрепи. 

Нещо повече. Нормативната уредба нерядко, вместо да ги насърчава и подкрепя да прилагат медицина, основана на доказателства, в някои случаи изисква от тях точно обратното. Защото алгоритъмът, към който те са длъжни да се придържат, е на клиничната пътека. Ако този алгоритъм не бъде изпълнен, Здравната каса няма да заплати лечението или ще глоби лечебното заведение. А тези алгоритми нямат отношение към медицината, основана на доказателства, и нерядко принуждават лекарите да извършват ненужни дейности, но и не им дават възможност да извършат необходимите. Системата не предполага субективните качества и умения на работещите в нея да бъдат възпитавани в духа на съвременната медицинска наука. И това е голям проблем за всички, не само за пациентите или само за медиците. 

Бихте ли обяснили понятието „медицина, основана на доказателства“? 

Добрият медицински специалист е този, който прилага утвърдените от науката и практиката методи, но ги прилага чрез позоваване на медицината, основана на доказателства. Това означава, че когато лекарят или медицинската сестра, или акушерката, или друг медицински специалист избира да прилага даден метод, той трябва да бъде актуален и да бъде доказан. В медицинската наука има много ясни правила как един диагностично-лечебен метод се утвърждава и медицинските специалисти трябва да познават тези правила. 

Нека се върнем към нашия си пример – ако се чудим дали прилагането на метода „Кристелер“ съответства на качествената медицинска помощ, можем да проверим, следвайки тези правила на медицинската наука. И ще установим как през 1967 г. се е смятало, че това е единственият начин в определени случаи да подпомогнеш раждането при даден проблем. Но през годините са събрани данни, проведени са научни изследвания и проучвания, вече е доказано, че рискът от прилагането на метода надхвърля в пъти повече евентуалните ползи. Затова към момента това не е утвърден от науката и практиката метод.

Как се гарантира на практика дължимата грижа и качеството на медицинската помощ в по-добре регулирани системи? 

Нито една система не е съвършена, но има системи, в които за това се мисли активно. Един от ключовите моменти е осигуряването на задължително продължаващо обучение за медицинските специалисти, което да бъде и ефективно, и контролирано. Казано по-просто, държавата трябва да гарантира, че тези хора периодично се обучават в новите достижения на медицинската наука в своята сфера, а след това да контролира как новостите се прилагат в практиката. 

Другото е създаването на работещи етични правила и подкрепянето на медицинските специалисти да се придържат към тях. Тук говорим за нещо просто, каквото е очакването на пациента да получи човешко отношение, защото той е уязвимата страна в отношенията с лекаря и защото никой не е станал пациент по желание. Да се създадат условия за такова общуване между медици и пациенти, които да гарантират, че информираното съгласие не е само една бланка, която пациентът подписва под натиск, а истинска комуникация, в която му става ясно какво му предстои.

Третото, за което се мисли, е гарантирането на пътя на пациента през системата. Това са взаимовръзките между различни специалисти, различни лечебни заведения и различни етапи на лечение. Наскоро при мен беше пациентка, диагностицирана със злокачествено заболяване на кръвта. В решението си онкологичната комисия препоръчваше ревизия на хистологичните блокчета, както и извършване на ядрено-магнитен резонанс. Без обаче никой да насочи жената къде да направи този ядрено-магнитен резонанс, по какъв начин да се снабди с хистологията си, къде да бъде осъществена тази ревизия, кой да я осъществи и след като евентуално направи ревизията и ядрено-магнитния резонанс, какво да прави с резултатите. Никаква представа. Това беше забавило лечението на пациентката с около три месеца. 

В изследването си стигнахте ли до извод какво може да направи един пациент, за да си осигури качествена медицинска помощ въпреки недостатъците и противоречията в правилата? 

За съжаление, смея да кажа, че обективни предпоставки на законодателно равнище, които да гарантират подобна дължима грижа, няма. Това е въпрос на субективните качества на лекарите или на субективните качества на ръководителя на лечебно заведение. Но за да внеса малко оптимизъм, трябва да кажа, че все по-често срещам млади лекари, които искат да работят по съвременен начин, инвестират сами в развитието си и се опитват да бъдат коректив на системните грешки. В повечето случаи и в повечето лечебни заведения наистина се полага старание да се оказва оптималната медицинска помощ, която всеки пациент заслужава да получи и има право да получи.

Explore cloud security in the age of generative AI at AWS re:Inforce 2024

Post Syndicated from Chris Betz original https://aws.amazon.com/blogs/security/explore-cloud-security-in-the-age-of-generative-ai-at-aws-reinforce-2024/

AWS re:Inforce 2024

As the Chief Information Security Officer (CISO) at AWS, I’m personally committed to helping security teams of all skill levels and sizes navigate security for generative artificial intelligence (AI). As a former AWS customer, I know the value of hands-on security learning and talking in-person to the people who build and run AWS security. That’s why I’m excited for you to join me at AWS re:Inforce 2024, our annual cloud security event, where you can collaborate with experts, partners, and the builders who are driving the future of security in the generative AI era.

Whether you want to build deep technical expertise, understand how to prioritize your security investments, or learn how to apply foundational security best practices, re:Inforce is a great opportunity to dive deep into the convergence of security and AI. AI and machine learning (ML) have been a focus for Amazon for more than 25 years. It’s inspiring to take advantage of and adapt to the impact generative AI is having on the world.

AWS re:Inforce isn’t just a conference—it’s a catalyst for innovation and collaboration in the realm of cloud security. This year, we’re coming together June 10–12 in Pennsylvania for 2.5 days of immersive cloud security learning designed to help drive your business initiatives. At AWS, we’ve always believed that security is a business enabler. Security reduces risk, reinforces resilience, and empowers confident innovation. Security helps organizations use new technologies such as generative AI quickly and safely, creating better experiences for customers, partners, and employees.

Here’s a bit of what you can expect:

How we secure AWS today and my vision for the future
AWS re:Inforce 2024 will begin with my keynote on Tuesday, June 11, 2024 at 9:00 AM EST. By the time re:Inforce comes around, I will have been the CISO of AWS for nearly a year. It’s incredible to think about all the innovations that have happened within AWS Security during that time. You’ll hear about these innovations, what we learned from them, and how we use them to secure AWS. I’ll also share my vision for the future of AWS Security. Steve Schmidt, CSO of Amazon, will take the stage to share his thoughts on how a strong culture of security supports the safe use of generative AI.

Navigate security for generative AI and other emerging trends
Now is the time for security teams to empower their builders to confidently innovate with generative AI. Be the first to hear about the latest AWS security advances around generative AI and have access to interactive sessions where you can learn how to implement AI workloads securely, explore use cases from other customers, and see demos from AWS and our partners of AI-driven security in action. Be sure to check out our Innovation Talks, where AWS security experts will give in-depth talks on essential topics including cryptography, generative AI, and building a culture of security.

This year, we’ve extended the event by half a day to give you more learning opportunities. At re:Inforce, you can personalize your agenda by choosing from more than 250 different sessions spanning data protection; identity and access management; threat detection and incident response; network and infrastructure security; governance, risk, and compliance; and application security. And we have an inspiring line-up of customer speakers this year who will share their firsthand experience innovating securely on AWS. More than 90 of our trusted security partners will also be there to help you simplify and integrate your security portfolio, and those of you looking to deepen security expertise will find more than 70 percent of re:Inforce sessions are advanced or expert level.

Take advantage of the latest innovations in AWS security
You’ll hear about our latest announcements and product launches and learn how to operationalize these security innovations in over 100 interactive-format sessions. In response to customer feedback, we’ve added more interactive session formats, including chalk talks, code talks, workshops, and builders’ sessions. Don’t miss this chance to connect directly with AWS experts to learn how you can get more out of the security tools you already have.

Connect with AWS experts, partners, and your peers
The opportunity to come together with other members of the security community is something that really energizes me. Coming together in person gives us all the opportunity to connect with peers, find mentors, and learn from each other. You can advance your career development goals with attendee-matchmaking tools, explore the Expo to connect with trusted partners, and join lounges and activities designed for specific interest groups in the security community.

Meet up with AWS Security Heroes, security professionals, IT leaders, educators, and developers with a shared mission to foster excellence in the practice and profession of cybersecurity and digital identity. Or, connect with members of the AWS Community Builders program, AWS technical enthusiasts and emerging thought leaders who are passionate about sharing knowledge and connecting with the security community. There’s no better opportunity to make new connections with the diverse AWS security community.

Register today with the code SECBLOfnakb to receive a limited time $150 USD discount, while supplies last.

If you’re in the first five years of your cloud security career, you might qualify for the All Builders Welcome Grant. This grant removes the financial barrier to attend AWS re:Inforce for underrepresented technologists as part of our commitment to creating a cybersecurity community that is inclusive, diverse, and equitable.

We’ll share additional details over the coming weeks on the AWS re:Inforce website, @awscloud, and @AWSSecurityInfo. I’m excited to welcome you all to AWS re:Inforce this June.

See you in Philadelphia!

Chris

Chris Betz

Chris is CISO at AWS. He oversees security teams and leads the development and implementation of security policies with the aim of managing risk and aligning the company’s security posture with business objectives. Chris joined Amazon in August 2023 after holding CISO and security leadership roles at leading companies. He lives in Northern Virginia with his family.

Securing generative AI: data, compliance, and privacy considerations

Post Syndicated from Mark Keating original https://aws.amazon.com/blogs/security/securing-generative-ai-data-compliance-and-privacy-considerations/

Generative artificial intelligence (AI) has captured the imagination of organizations and individuals around the world, and many have already adopted it to help improve workforce productivity, transform customer experiences, and more.

When you use a generative AI-based service, you should understand how the information that you enter into the application is stored, processed, shared, and used by the model provider or the provider of the environment that the model runs in. Organizations that offer generative AI solutions have a responsibility to their users and consumers to build appropriate safeguards, designed to help verify privacy, compliance, and security in their applications and in how they use and train their models.

This post continues our series on how to secure generative AI, and provides guidance on the regulatory, privacy, and compliance challenges of deploying and building generative AI workloads. We recommend that you start by reading the first post of this series: Securing generative AI: An introduction to the Generative AI Security Scoping Matrix, which introduces you to the Generative AI Scoping Matrix—a tool to help you identify your generative AI use case—and lays the foundation for the rest of our series.

Figure 1 shows the scoping matrix:

Figure 1: Generative AI Scoping Matrix

Figure 1: Generative AI Scoping Matrix

Broadly speaking, we can classify the use cases in the scoping matrix into two categories: prebuilt generative AI applications (Scopes 1 and 2), and self-built generative AI applications (Scopes 3–5). Although some consistent legal, governance, and compliance requirements apply to all five scopes, each scope also has unique requirements and considerations. We will cover some key considerations and best practices for each scope.

Scope 1: Consumer applications

Consumer applications are typically aimed at home or non-professional users, and they’re usually accessed through a web browser or a mobile app. Many applications that created the initial excitement around generative AI fall into this scope, and can be free or paid for, using a standard end-user license agreement (EULA). Although they might not be built specifically for enterprise use, these applications have widespread popularity. Your employees might be using them for their own personal use and might expect to have such capabilities to help with work tasks.

Many large organizations consider these applications to be a risk because they can’t control what happens to the data that is input or who has access to it. In response, they ban Scope 1 applications. Although we encourage due diligence in assessing the risks, outright bans can be counterproductive. Banning Scope 1 applications can cause unintended consequences similar to that of shadow IT, such as employees using personal devices to bypass controls that limit use, reducing visibility into the applications that they use. Instead of banning generative AI applications, organizations should consider which, if any, of these applications can be used effectively by the workforce, but within the bounds of what the organization can control, and the data that are permitted for use within them.

To help your workforce understand the risks associated with generative AI and what is acceptable use, you should create a generative AI governance strategy, with specific usage guidelines, and verify your users are made aware of these policies at the right time. For example, you could have a proxy or cloud access security broker (CASB) control that, when accessing a generative AI based service, provides a link to your company’s public generative AI usage policy and a button that requires them to accept the policy each time they access a Scope 1 service through a web browser when using a device that your organization issued and manages. This helps verify that your workforce is trained and understands the risks, and accepts the policy before using such a service.

To help address some key risks associated with Scope 1 applications, prioritize the following considerations:

  • Identify which generative AI services your staff are currently using and seeking to use.
  • Understand the service provider’s terms of service and privacy policy for each service, including who has access to the data and what can be done with the data, including prompts and outputs, how the data might be used, and where it’s stored.
  • Understand the source data used by the model provider to train the model. How do you know the outputs are accurate and relevant to your request? Consider implementing a human-based testing process to help review and validate that the output is accurate and relevant to your use case, and provide mechanisms to gather feedback from users on accuracy and relevance to help improve responses.
  • Seek legal guidance about the implications of the output received or the use of outputs commercially. Determine who owns the output from a Scope 1 generative AI application, and who is liable if the output uses (for example) private or copyrighted information during inference that is then used to create the output that your organization uses.
  • The EULA and privacy policy of these applications will change over time with minimal notice. Changes in license terms can result in changes to ownership of outputs, changes to processing and handling of your data, or even liability changes on the use of outputs. Create a plan/strategy/mechanism to monitor the policies on approved generative AI applications. Review the changes and adjust your use of the applications accordingly.

For Scope 1 applications, the best approach is to consider the input prompts and generated content as public, and not to use personally identifiable information (PII), highly sensitive, confidential, proprietary, or company intellectual property (IP) data with these applications.

Scope 1 applications typically offer the fewest options in terms of data residency and jurisdiction, especially if your staff are using them in a free or low-cost price tier. If your organization has strict requirements around the countries where data is stored and the laws that apply to data processing, Scope 1 applications offer the fewest controls, and might not be able to meet your requirements.

Scope 2: Enterprise applications

The main difference between Scope 1 and Scope 2 applications is that Scope 2 applications provide the opportunity to negotiate contractual terms and establish a formal business-to-business (B2B) relationship. They are aimed at organizations for professional use with defined service level agreements (SLAs) and licensing terms and conditions, and they are usually paid for under enterprise agreements or standard business contract terms. The enterprise agreement in place usually limits approved use to specific types (and sensitivities) of data.

Most aspects from Scope 1 apply to Scope 2. However, in Scope 2, you are intentionally using your proprietary data and encouraging the widespread use of the service across your organization. When assessing the risk, consider these additional points:

  • Determine the acceptable classification of data that is permitted to be used with each Scope 2 application, update your data handling policy to reflect this, and include it in your workforce training.
  • Understand the data flow of the service. Ask the provider how they process and store your data, prompts, and outputs, who has access to it, and for what purpose. Do they have any certifications or attestations that provide evidence of what they claim and are these aligned with what your organization requires. Make sure that these details are included in the contractual terms and conditions that you or your organization agree to.
  • What (if any) data residency requirements do you have for the types of data being used with this application? Understand where your data will reside and if this aligns with your legal or regulatory obligations.
    • Many major generative AI vendors operate in the USA. If you are based outside the USA and you use their services, you have to consider the legal implications and privacy obligations related to data transfers to and from the USA.
    • Vendors that offer choices in data residency often have specific mechanisms you must use to have your data processed in a specific jurisdiction. You might need to indicate a preference at account creation time, opt into a specific kind of processing after you have created your account, or connect to specific regional endpoints to access their service.
  • Most Scope 2 providers want to use your data to enhance and train their foundational models. You will probably consent by default when you accept their terms and conditions. Consider whether that use of your data is permissible. If your data is used to train their model, there is a risk that a later, different user of the same service could receive your data in their output. If you need to prevent reuse of your data, find the opt-out options for your provider. You might need to negotiate with them if they don’t have a self-service option for opting out.
  • When you use an enterprise generative AI tool, your company’s usage of the tool is typically metered by API calls. That is, you pay a certain fee for a certain number of calls to the APIs. Those API calls are authenticated by the API keys the provider issues to you. You need to have strong mechanisms for protecting those API keys and for monitoring their usage. If the API keys are disclosed to unauthorized parties, those parties will be able to make API calls that are billed to you. Usage by those unauthorized parties will also be attributed to your organization, potentially training the model (if you’ve agreed to that) and impacting subsequent uses of the service by polluting the model with irrelevant or malicious data.

Scope 3: Pre-trained models

In contrast to prebuilt applications (Scopes 1 and 2), Scope 3 applications involve building your own generative AI applications by using a pretrained foundation model available through services such as Amazon Bedrock and Amazon SageMaker JumpStart. You can use these solutions for your workforce or external customers. Much of the guidance for Scopes 1 and 2 also applies here; however, there are some additional considerations:

  • A common feature of model providers is to allow you to provide feedback to them when the outputs don’t match your expectations. Does the model vendor have a feedback mechanism that you can use? If so, make sure that you have a mechanism to remove sensitive content before sending feedback to them.
  • Does the provider have an indemnification policy in the event of legal challenges for potential copyright content generated that you use commercially, and has there been case precedent around it?
  • Is your data included in prompts or responses that the model provider uses? If so, for what purpose and in which location, how is it protected, and can you opt out of the provider using it for other purposes, such as training? At Amazon, we don’t use your prompts and outputs to train or improve the underlying models in Amazon Bedrock and SageMaker JumpStart (including those from third parties), and humans won’t review them. Also, we don’t share your data with third-party model providers. Your data remains private to you within your AWS accounts.
  • Establish a process, guidelines, and tooling for output validation. How do you make sure that the right information is included in the outputs based on your fine-tuned model, and how do you test the model’s accuracy? For example:
    • If the application is generating text, create a test and output validation process that is tested by humans on a regular basis (for example, once a week) to verify the generated outputs are producing the expected results.
    • Another approach could be to implement a feedback mechanism that the users of your application can use to submit information on the accuracy and relevance of output.
    • If generating programming code, this should be scanned and validated in the same way that any other code is checked and validated in your organization.

Scope 4: Fine-tuned models

Scope 4 is an extension of Scope 3, where the model that you use in your application is fine-tuned with data that you provide to improve its responses and be more specific to your needs. The considerations for Scope 3 are also relevant to Scope 4; in addition, you should consider the following:

  • What is the source of the data used to fine-tune the model? Understand the quality of the source data used for fine-tuning, who owns it, and how that could lead to potential copyright or privacy challenges when used.
  • Remember that fine-tuned models inherit the data classification of the whole of the data involved, including the data that you use for fine-tuning. If you use sensitive data, then you should restrict access to the model and generated content to that of the classified data.
  • As a general rule, be careful what data you use to tune the model, because changing your mind will increase cost and delays. If you tune a model on PII directly, and later determine that you need to remove that data from the model, you can’t directly delete data. With current technology, the only way for a model to unlearn data is to completely retrain the model. Retraining usually requires a lot of time and money.

Scope 5: Self-trained models

With Scope 5 applications, you not only build the application, but you also train a model from scratch by using training data that you have collected and have access to. Currently, this is the only approach that provides full information about the body of data that the model uses. The data can be internal organization data, public data, or both. You control many aspects of the training process, and optionally, the fine-tuning process. Depending on the volume of data and the size and complexity of your model, building a scope 5 application requires more expertise, money, and time than any other kind of AI application. Although some customers have a definite need to create Scope 5 applications, we see many builders opting for Scope 3 or 4 solutions.

For Scope 5 applications, here are some items to consider:

  • You are the model provider and must assume the responsibility to clearly communicate to the model users how the data will be used, stored, and maintained through a EULA.
  • Unless required by your application, avoid training a model on PII or highly sensitive data directly.
  • Your trained model is subject to all the same regulatory requirements as the source training data. Govern and protect the training data and trained model according to your regulatory and compliance requirements. After the model is trained, it inherits the data classification of the data that it was trained on.
  • To limit potential risk of sensitive information disclosure, limit the use and storage of the application users’ data (prompts and outputs) to the minimum needed.

AI regulation and legislation

AI regulations are rapidly evolving and this could impact you and your development of new services that include AI as a component of the workload. At AWS, we’re committed to developing AI responsibly and taking a people-centric approach that prioritizes education, science, and our customers, to integrate responsible AI across the end-to-end AI lifecycle. For more details, see our Responsible AI resources. To help you understand various AI policies and regulations, the OECD AI Policy Observatory is a good starting point for information about AI policy initiatives from around the world that might affect you and your customers. At the time of publication of this post, there are over 1,000 initiatives across more 69 countries.

In this section, we consider regulatory themes from two different proposals to legislate AI: the European Union (EU) Artificial Intelligence (AI) Act (EUAIA), and the United States Executive Order on Artificial Intelligence.

Our recommendation for AI regulation and legislation is simple: monitor your regulatory environment, and be ready to pivot your project scope if required.

Theme 1: Data privacy

According to the UK Information Commissioners Office (UK ICO), the emergence of generative AI doesn’t change the principles of data privacy laws, or your obligations to uphold them. There are implications when using personal data in generative AI workloads. Personal data might be included in the model when it’s trained, submitted to the AI system as an input, or produced by the AI system as an output. Personal data from inputs and outputs can be used to help make the model more accurate over time via retraining.

For AI projects, many data privacy laws require you to minimize the data being used to what is strictly necessary to get the job done. To go deeper on this topic, you can use the eight questions framework published by the UK ICO as a guide. We recommend using this framework as a mechanism to review your AI project data privacy risks, working with your legal counsel or Data Protection Officer.

In simple terms, follow the maxim “don’t record unnecessary data” in your project.

Theme 2: Transparency and explainability

The OECD AI Observatory defines transparency and explainability in the context of AI workloads. First, it means disclosing when AI is used. For example, if a user interacts with an AI chatbot, tell them that. Second, it means enabling people to understand how the AI system was developed and trained, and how it operates. For example, the UK ICO provides guidance on what documentation and other artifacts you should provide that describe how your AI system works. In general, transparency doesn’t extend to disclosure of proprietary sources, code, or datasets. Explainability means enabling the people affected, and your regulators, to understand how your AI system arrived at the decision that it did. For example, if a user receives an output that they don’t agree with, then they should be able to challenge it.

So what can you do to meet these legal requirements? In practical terms, you might be required to show the regulator that you have documented how you implemented the AI principles throughout the development and operation lifecycle of your AI system. In addition to the ICO guidance, you can also consider implementing an AI Management system based on ISO42001:2023.

Diving deeper on transparency, you might need to be able to show the regulator evidence of how you collected the data, as well as how you trained your model.

Transparency with your data collection process is important to reduce risks associated with data. One of the leading tools to help you manage the transparency of the data collection process in your project is Pushkarna and Zaldivar’s Data Cards (2022) documentation framework. The Data Cards tool provides structured summaries of machine learning (ML) data; it records data sources, data collection methods, training and evaluation methods, intended use, and decisions that affect model performance. If you import datasets from open source or public sources, review the Data Provenance Explorer initiative. This project has audited over 1,800 datasets for licensing, creators, and origin of data.

Transparency with your model creation process is important to reduce risks associated with explainability, governance, and reporting. Amazon SageMaker has a feature called Model Cards that you can use to help document critical details about your ML models in a single place, and streamlining governance and reporting. You should catalog details such as intended use of the model, risk rating, training details and metrics, and evaluation results and observations.

When you use models that were trained outside of your organization, then you will need to rely on Standard Contractual Clauses. SCC’s enable sharing and transfer of any personal information that the model might have been trained on, especially if data is being transferred from the EU to third countries. As part of your due diligence, you should contact the vendor of your model to ask for a Data Card, Model Card, Data Protection Impact Assessment (for example, ISO29134:2023), or Transfer Impact Assessment (for example, IAPP). If no such documentation exists, then you should factor this into your own risk assessment when making a decision to use that model. Two examples of third-party AI providers that have worked to establish transparency for their products are Twilio and SalesForce. Twilio provides AI Nutrition Facts labels for its products to make it simple to understand the data and model. SalesForce addresses this challenge by making changes to their acceptable use policy.

Theme 3: Automated decision making and human oversight

The final draft of the EUAIA, which starts to come into force from 2026, addresses the risk that automated decision making is potentially harmful to data subjects because there is no human intervention or right of appeal with an AI model. Responses from a model have a likelihood of accuracy, so you should consider how to implement human intervention to increase certainty. This is important for workloads that can have serious social and legal consequences for people—for example, models that profile people or make decisions about access to social benefits. We recommend that when you are developing your business case for an AI project, consider where human oversight should be applied in the workflow.

The UK ICO provides guidance on what specific measures you should take in your workload. You might give users information about the processing of the data, introduce simple ways for them to request human intervention or challenge a decision, carry out regular checks to make sure that the systems are working as intended, and give individuals the right to contest a decision.

The US Executive Order for AI describes the need to protect people from automatic discrimination based on sensitive characteristics. The order places the onus on the creators of AI products to take proactive and verifiable steps to help verify that individual rights are protected, and the outputs of these systems are equitable.

Prescriptive guidance on this topic would be to assess the risk classification of your workload and determine points in the workflow where a human operator needs to approve or check a result. Addressing bias in the training data or decision making of AI might include having a policy of treating AI decisions as advisory, and training human operators to recognize those biases and take manual actions as part of the workflow.

Theme 4: Regulatory classification of AI systems

Just like businesses classify data to manage risks, some regulatory frameworks classify AI systems. It is a good idea to become familiar with the classifications that might affect you. The EUAIA uses a pyramid of risks model to classify workload types. If a workload has an unacceptable risk (according to the EUAIA), then it might be banned altogether.

Banned workloads
The EUAIA identifies several AI workloads that are banned, including CCTV or mass surveillance systems, systems used for social scoring by public authorities, and workloads that profile users based on sensitive characteristics. We recommend you perform a legal assessment of your workload early in the development lifecycle using the latest information from regulators.

High risk workloads
There are also several types of data processing activities that the Data Privacy law considers to be high risk. If you are building workloads in this category then you should expect a higher level of scrutiny by regulators, and you should factor extra resources into your project timeline to meet regulatory requirements. The good news is that the artifacts you created to document transparency, explainability, and your risk assessment or threat model, might help you meet the reporting requirements. To see an example of these artifacts. see the AI and data protection risk toolkit published by the UK ICO.

Examples of high-risk processing include innovative technology such as wearables, autonomous vehicles, or workloads that might deny service to users such as credit checking or insurance quotes. We recommend that you engage your legal counsel early in your AI project to review your workload and advise on which regulatory artifacts need to be created and maintained. You can see further examples of high risk workloads at the UK ICO site here.

The EUAIA also pays particular attention to profiling workloads. The UK ICO defines this as “any form of automated processing of personal data consisting of the use of personal data to evaluate certain personal aspects relating to a natural person, in particular to analyse or predict aspects concerning that natural person’s performance at work, economic situation, health, personal preferences, interests, reliability, behaviour, location or movements.” Our guidance is that you should engage your legal team to perform a review early in your AI projects.

We recommend that you factor a regulatory review into your timeline to help you make a decision about whether your project is within your organization’s risk appetite. We recommend you maintain ongoing monitoring of your legal environment as the laws are rapidly evolving.

Theme 6: Safety

ISO42001:2023 defines safety of AI systems as “systems behaving in expected ways under any circumstances without endangering human life, health, property or the environment.”

The United States AI Bill of Rights states that people have a right to be protected from unsafe or ineffective systems. In October 2023, President Biden issued the Executive Order on Safe, Secure and Trustworthy Artificial Intelligence, which highlights the requirement to understand the context of use for an AI system, engaging the stakeholders in the community that will be affected by its use. The Executive Order also describes the documentation, controls, testing, and independent validation of AI systems, which aligns closely with the explainability theme that we discussed previously. For your workload, make sure that you have met the explainability and transparency requirements so that you have artifacts to show a regulator if concerns about safety arise. The OECD also offers prescriptive guidance here, highlighting the need for traceability in your workload as well as regular, adequate risk assessments—for example, ISO23894:2023 AI Guidance on risk management.

Conclusion

Although generative AI might be a new technology for your organization, many of the existing governance, compliance, and privacy frameworks that we use today in other domains apply to generative AI applications. Data that you use to train generative AI models, prompt inputs, and the outputs from the application should be treated no differently to other data in your environment and should fall within the scope of your existing data governance and data handling policies. Be mindful of the restrictions around personal data, especially if children or vulnerable people can be impacted by your workload. When fine-tuning a model with your own data, review the data that is used and know the classification of the data, how and where it’s stored and protected, who has access to the data and trained models, and which data can be viewed by the end user. Create a program to train users on the uses of generative AI, how it will be used, and data protection policies that they need to adhere to. For data that you obtain from third parties, make a risk assessment of those suppliers and look for Data Cards to help ascertain the provenance of the data.

Regulation and legislation typically take time to formulate and establish; however, existing laws already apply to generative AI, and other laws on AI are evolving to include generative AI. Your legal counsel should help keep you updated on these changes. When you build your own application, you should be aware of new legislation and regulation that is in draft form (such as the EU AI Act) and whether it will affect you, in addition to the many others that might already exist in locations where you operate, because they could restrict or even prohibit your application, depending on the risk the application poses.

At AWS, we make it simpler to realize the business value of generative AI in your organization, so that you can reinvent customer experiences, enhance productivity, and accelerate growth with generative AI. If you want to dive deeper into additional areas of generative AI security, check out the other posts in our Securing Generative AI series:

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Generative AI on AWS re:Post or contact AWS Support.

Mark Keating

Mark Keating

Mark is an AWS Security Solutions Architect based in the UK who works with global healthcare and life sciences and automotive customers to solve their security and compliance challenges and help them reduce risk. He has over 20 years of experience working with technology, within operations, solutions, and enterprise architecture roles.

Samuel Waymouth

Samuel Waymouth

Samuel is a Senior Security and Compliance Solutions Architect on the AWS Industries team. He works with customers and partners to help demystify regulation, IT standards, risk management, control mapping, and how to apply these with AWS service features. Outside of work, he enjoys Tae Kwon-do, motorcycles, traveling, playing guitar, experimenting with microcontrollers and IoT, and spending time with family.

How to use OAuth 2.0 in Amazon Cognito: Learn about the different OAuth 2.0 grants

Post Syndicated from Prashob Krishnan original https://aws.amazon.com/blogs/security/how-to-use-oauth-2-0-in-amazon-cognito-learn-about-the-different-oauth-2-0-grants/

Implementing authentication and authorization mechanisms in modern applications can be challenging, especially when dealing with various client types and use cases. As developers, we often struggle to choose the right authentication flow to balance security, user experience, and application requirements. This is where understanding the OAuth 2.0 grant types comes into play. Whether you’re building a traditional web application, a mobile app, or a machine-to-machine communication system, understanding the OAuth 2.0 grant types can help you implement robust and secure authentication and authorization mechanism.

In this blog post, we show you the different OAuth 2.0 grants and how to implement them in Amazon Cognito. We review the purpose of each grant, their relevance in modern application development, and which grant is best suited for different application requirements.

OAuth 2.0 is an authorization framework that enables secure and seamless access to resources on behalf of users without the need to share sensitive credentials. The primary objective of OAuth 2.0 is to establish a secure, delegated, and scoped access mechanism that allows third-party applications to interact with user data while maintaining robust privacy and security measures.

OpenID Connect, often referred to as OIDC, is a protocol based on OAuth 2.0. It extends OAuth 2.0 to provide user authentication, identity verification, and user information retrieval. OIDC is a crucial component for building secure and user-friendly authentication experiences in applications. Amazon Cognito supports OIDC, meaning it supports user authentication and identity verification according to OIDC standards.

Amazon Cognito is an identity environment for web and mobile applications. Its two main components are user pools and identity pools. A Cognito user pool is a user directory, an authentication server, and an authorization service for OAuth 2.0 tokens. With it, you can authenticate and authorize users natively or from a federated identity such as your enterprise directory, or from consumer identity providers such as Google or Facebook. Cognito Identity Pool can exchange OAuth 2.0 tokens (among other options) for AWS credentials.

Implementing OAuth 2.0 grants using Amazon Cognito

The OAuth 2.0 standard defines four main roles; these are important to know as we discuss the grants:

  • A resource owner owns the data in the resource server and can grant access to the resource (such as a database admin).
  • A resource server hosts the protected resources that the application wants to access (such as a SQL server).
  • A client is an application making requests for the protected resources on behalf of the resource owner and with its authorization (such as an analytics application).
  • An authorization server is a server that issues scoped tokens after the user is authenticated and has consented to the issuance of the token under the desired scope (such as Amazon Cognito).

A few other useful concepts before we dive into the OAuth 2.0 grants:

  • Access tokens are at the core of OAuth 2.0’s operation. These tokens are short-lived credentials that the client application uses to prove its authorized status when requesting resources from the resource server. Additionally, OAuth 2.0 might involve the use of refresh tokens, which provide a mechanism for clients to obtain new access tokens without requiring the resource owner’s intervention.
  • An ID token is a JSON Web Token (JWT) introduced by OpenID Connect that contains information about the authentication event of the user. They allow applications to verify the identity of the user, make informed decisions about the user’s authentication status, and personalize the user’s experience.
  • A scope is a level of access that an application can request to a resource. Scopes define the specific permissions that a client application can request when obtaining an access token. You can use scopes to fine-tune the level of access granted to the client. For example, an OAuth 2.0 request might include the scope read:profile, indicating that the client application is requesting read-only access to the user’s profile information. Another request might include the scope write:photos, indicating the client’s need to write to the user’s photo collection. In Amazon Cognito, you can define custom scopes along with standard OAuth 2.0 scopes such as openid, profile, email, or phone to align with your application’s requirements. You can use this flexibility to manage access permissions efficiently and securely.

A typical high-level OAuth 2.0 flow looks like the Figure 1:

Figure 1: OAuth 2.0 flow

Figure 1: OAuth 2.0 flow

Below are the steps involved in the OAuth 2.0 flow

  1. The client requests authorization from the resource owner. This is done through the authorization server (Amazon Cognito) as an intermediary.
  2. The resource owner provides the authorization grant to the client. This can be one of the many grant types, which are discussed in detail in the next paragraph. The type of grant used depends on the method used by the client to request authorization from the resource owner.
  3. The client requests an access token by authenticating with Cognito.
  4. Cognito authenticates the client (the authentication method based on the grant type) and issues an access token if the authorization is valid.
  5. The access token is presented to the resource server as the client requests the protected resource.
  6. The resource server checks the access token’s signature and attributes and serves the request if it is valid.

There are several different grant types, four of which are described in the following sections.

Authorization code grant

The authorization code grant type is used by clients to securely exchange an authorization code for an access token. It’s used by both web applications and native applications to get an access token after a user authenticates to an application. After the user returns to the client through the redirect URI (the URL where the authentication server redirects the browser after it authorizes the user), the application gets the authorization code from the URL and uses it to request an access token.

This grant type is suitable for general cases as only one authentication flow is used, regardless of what operation is performed or who is performing it. This grant is considered secure as it requests an access token with a single-use code instead of exposing the actual access tokens. This helps prevent the application from potentially accessing user credentials.

Figure 2: Authorization code grant flow

Figure 2: Authorization code grant flow

Below are the steps involved in the authorization code grant flow

  1. The process begins with the client initiating the sequence, directing the user-agent (that is, the browser) of the resource owner to the authorization endpoint. In this action, the client provides its client identifier, the scope it’s requesting, a local state, and a redirection URI to which the authorization server (Amazon Cognito) will return the user agent after either granting or denying access.
  2. Cognito authenticates the resource owner (through the user agent) and establishes whether the resource owner grants or denies the client’s access request using user pool authentication.
  3. Cognito redirects the user agent back to the client using the redirection URI that was provided in step (1) with an authorization code in the query string (such as http://www.example.com/webpage?code=<authcode>).
  4. The client requests an access token from the Cognito’s token endpoint by including the authorization code received in step (3). When making the request, the client authenticates with the Cognito typically with a client ID and a secret. The client includes the redirection URI used to obtain the authorization code for verification.
  5. Cognito authenticates the client, validates the authorization code, and makes sure that the redirection URI received matches the URI used to redirect the client in step (3). If valid, Cognito responds with an access token.

An implementation of the authorization code grant using Amazon Cognito looks like the following:

  1. An application makes an HTTP GET request to _DOMAIN/oauth2/authorize, where AUTH_DOMAIN represents the user pool’s configured domain. This request includes the following query parameters:
    • response_type – Set to code for this grant type.
    • client_id – The ID for the desired user pool app client.
    • redirect_uri – The URL that a user is directed to after successful authentication.
    • state (optional but recommended) – A random value that’s used to prevent cross-site request forgery (CSRF) attacks.
    • scope (optional) – A space-separated list of scopes to request for the generated tokens. Note that:
      • An ID token is only generated if the openid scope is requested.
      • The phone, email, and profile scopes can only be requested if openid is also requested.
      • A vended access token can only be used to make user pool API calls if aws.cognito.signin.user.admin (user pool’s reserved API scope) is requested.
    • identity_provider (optional) – Indicates the provider that the end user should authenticate with.
    • idp_identifier (optional) – Same as identity_provider but doesn’t expose the provider’s real name.
    • nonce (optional) – A random value that you can add to the request. The nonce value that you provide is included in the ID token that Amazon Cognito issues. To guard against replay attacks, your app can inspect the nonce claim in the ID token and compare it to the one you generated. For more information about the nonce claim, see ID token validation in the OpenID Connect standard.
  2. A CSRF token is returned in a cookie. If an identity provider was specified in the request from step 1, the rest of this step is skipped. The user is automatically redirected to the appropriate identity provider’s authentication page. Otherwise, the user is redirected to https://AUTH_DOMAIN/login (which hosts the auto-generated UI) with the same query parameters set from step 1. They can then either authenticate with the user pool or select one of the third-party providers that’s configured for the designated app client.
  3. The user authenticates with their identity provider through one of the following means:
    1. If the user uses the native user pool to authenticate, the hosted UI submits the user’s credentials through a POST request to https://AUTH_DOMAIN/login (including the original query parameters) along with some additional metadata.
    2. If the user selects a different identity provider to authenticate with, the user is redirected to that identity provider’s authentication page. After successful authentication the provider redirects the user to https://AUTH_DOMAIN/saml2/idpresponse with either an authorization token in the code query parameter or a SAML assertion in a POST request.
  4. After Amazon Cognito verifies the user pool credentials or provider tokens it receives, the user is redirected to the URL that was specified in the original redirect_uri query parameter. The redirect also sets a code query parameter that specifies the authorization code that was vended to the user by Cognito.
  5. The custom application that’s hosted at the redirect URL can then extract the authorization code from the query parameters and exchange it for user pool tokens. The exchange occurs by submitting a POST request to https://AUTH_DOMAIN/oauth2/token with the following application/x-www-form-urlencoded parameters:
    • grant_type – Set to authorization_code for this grant.
    • code – The authorization code that’s vended to the user.
    • client_id – Same as from the request in step 1.
    • redirect_uri – Same as from the request in step 1.

If the client application that was configured with a secret, the Authorization header for this request is set as Basic BASE64(CLIENT_ID:CLIENT_SECRET), where BASE64(CLIENT_ID:CLIENT_SECRET) is the base64 representation of the application client ID and application client secret, concatenated with a colon.

The JSON returned in the resulting response has the following keys:

  • access_token – A valid user pool access token.
  • refresh_token – A valid user pool refresh token. This can be used to retrieve new tokens by sending it through a POST request to https://AUTH_DOMAIN/oauth2/token, specifying the refresh_token and client_id parameters, and setting the grant_type parameter to refresh_token.
  • id_token – A valid user pool ID token. Note that an ID token is only provided if the openid scope was requested.
  • expires_in – The length of time (in seconds) that the provided ID or access tokens are valid.
  • token_type – Set to Bearer.

Here are some of the best practices to be followed when using the authorization code grant:

  • Use the Proof Key for Code Exchange (PKCE) extension with the authorization code grant, especially for public clients such as a single page web application. This is discussed in more detail in the following section.
  • Regularly rotate client secrets and credentials to minimize the risk of unauthorized access.
  • Implement session management to handle user sessions securely. This involves managing access token lifetimes, storing tokens, rotating refresh tokens, implementing token revocations and providing easy logout mechanisms that invalidate access and refresh tokens on user’s devices.

Authorization code grant with PKCE

To enhance security when using the authorization code grant, especially in public clients such as native applications, the PKCE extension was introduced. PKCE adds an extra layer of protection by making sure that only the client that initiated the authorization process can exchange the received authorization code for an access token. This combination is sometimes referred to as a PKCE grant.

It introduces a secret called the code verifier, which is a random value created by the client for each authorization request. This value is then hashed using a transformation method such as SHA256—this is now called the code challenge. The same steps are followed as the flow from Figure 2, however the code challenge is now added to the query string for the request to the authorization server (Amazon Cognito). The authorization server stores this code challenge for verification after the authentication process and redirects back with an authorization code. This authorization code along with the code verifier is sent to the authorization server, which then compares the previously stored code challenge with the code verifier. Access tokens are issued after the verification is successfully completed. Figure 3 outlines this process.

Figure 3: Authorization code grant flow with PKCE

Figure 3: Authorization code grant flow with PKCE

Authorization code grant with PKCE implementation is identical to authorization code grant except that Step 1 requires two additional query parameters:

  • code_challenge – The hashed, base64 URL-encoded representation of a random code that’s generated client side (code verifier). It serves as a PKCE, which mitigates bad actors from being able to use intercepted authorization codes.
  • code_challenge_method – The hash algorithm that’s used to generate the code_challenge. Amazon Cognito currently only supports setting this parameter to S256. This indicates that the code_challenge parameter was generated using SHA-256.

In step 5, when exchanging the authorization code with the user pool token, include an additional parameter:

  • code_verifier – The base64 URL-encoded representation of the unhashed, random string that was used to generate the PKCE code_challenge in the original request.

Implicit grant (not recommended)

Implicit grant was an OAuth 2.0 authentication grant type that allowed clients such as single-page applications and mobile apps to obtain user access tokens directly from the authorization endpoint. The grant type was implicit because no intermediate credentials (such as an authorization code) were issued and later used to obtain an access token. The implicit grant has been deprecated and it’s recommended that you use authorization code grant with PKCE instead. An effect of using the implicit grant was that it exposed access tokens directly in the URL fragment, which could potentially be saved in the browser history, intercepted, or exposed to other applications residing on the same device.

Figure 4: Implicit grant flow

Figure 4: Implicit grant flow

The implicit grant flow was designed to enable public client-side applications—such as single-page applications or mobile apps without a backend server component—to exchange authorization codes for tokens.

Steps 1, 2, and 3 of the implicit grant are identical to the authorization code grant steps, except that the response_type query parameter is set to token. Additionally, while a PKCE challenge can technically be passed, it isn’t used because the /oauth2/token endpoint is never accessed. The subsequent steps—starting with step 4—are as follows:

  1. After Amazon Cognito verifies the user pool credentials or provider tokens it receives, the user is redirected to the URL that was specified in the original redirect_uri query parameter. The redirect also sets the following query parameters:
    • access_token – A valid user pool access token.
    • expires_in – The length of time (in seconds) that the provided ID or access tokens are valid for.
    • token_type – Set to Bearer.
    • id_token – A valid user pool ID token. Note that an ID token is only provided if the openid scope was requested.

    Note that no refresh token is returned during an implicit grant, as specified in the RFC standard.

  2. The custom application that’s hosted at the redirect URL can then extract the access token and ID token (if they’re present) from the query parameters.

Here are some best practices for implicit grant:

  • Make access token lifetimes short. Implicit grant tokens can’t be revoked, so expiry is the only way to end their validity.
  • Implicit grant type is deprecated and should be used only for scenarios where a backend server component can’t be implemented, such as browser-based applications.

Client credentials grant

The client credentials grant is for machine-to-machine authentication. For example, a third-party application must verify its identity before it can access your system. The client can request an access token using only its client credentials (or other supported means of authentication) when the client is requesting access to the protected resources under its control or those of another resource owner that have been previously arranged with the authorization server.

The client credentials grant type must be used only by confidential clients. This means the client must have the ability to protect a secret string from users. Note that to use the client credentials grant, the corresponding user pool app client must have an associated app client secret.

Figure 5: Client credentials grant

Figure 5: Client credentials grant

The flow illustrated in Figure 5 includes the following steps:

  1. The client authenticates with the authorization server using a client ID and secret and requests an access token from the token endpoint.
  2. The authorization server authenticates the client, and if valid, issues an access token.

The detailed steps for the process are as follows:

  1. The application makes a POST request to https://AUTH_DOMAIN/oauth2/token, and specifies the following parameters:
    • grant_type – Set to client_credentials for this grant type.
    • client_id – The ID for the desired user pool app client.
    • scope – A space-separated list of scopes to request for the generated access token. Note that you can only use a custom scope with the client credentials grant.

In order to indicate that the application is authorized to make the request, the Authorization header for this request is set as Basic BASE64(CLIENT_ID:CLIENT_SECRET), where BASE64(CLIENT_ID:CLIENT_SECRET) is the base64 representation of the client ID and client secret, concatenated with a colon.

The Amazon Cognito authorization server returns a JSON object with the following keys:

  • access_token – A valid user pool access token.
  • expires_in – The length of time (in seconds) that the provided access token is valid.
  • token_type – Set to Bearer.

Note that, for this grant type, an ID token and a refresh token aren’t returned.

  1. The application uses the access token to make requests to an associated resource server.
  2. The resource server validates the received token and, if everything checks out, processes the request from the app.

Following are a few recommended practices while using the client credentials grant:

  • Store client credentials securely and avoid hardcoding them in your application. Use appropriate credential management practices, such as environment variables or secret management services.
  • Limit use cases. The client credentials grant is suitable for machine-to-machine authentication in highly trusted scenarios. Limit its use to cases where other grant types are not applicable.

Extension grant

Extension grants are a way to add support for non-standard token issuance scenarios such as token translation, delegation, or custom credentials. It lets you exchange access tokens from a third-party OAuth 2.0 authorization service with access tokens from Amazon Cognito. By defining the grant type using an absolute URI (determined by the authorization server) as the value of the grant_type argument of the token endpoint, and by adding other parameters required, the client can use an extension grant type.

An example of an extension grant is OAuth 2.0 device authorization grant (RFC 8628). This authorization grant makes it possible for internet-connected devices with limited input capabilities or that lack a user-friendly browser (such as wearables, smart assistants, video-streaming devices, smart-home automation, and health or medical devices) to review the authorization request on a secondary device, such as a smartphone, that has more advanced input and browser capabilities.

Some of the best practices to be followed when deciding to use extension grants are:

  • Extension grants are for non-standard token issuance scenarios. Use them only when necessary, and thoroughly document their use and purpose.
  • Conduct security audits and code reviews when implementing Extension grants to identify potential vulnerabilities and mitigate risks.

While Amazon Cognito doesn’t natively support extension grants currently, here is an example implementation of OAuth 2.0 device grant flow using AWS Lambda and Amazon DynamoDB.

Conclusion

In this blog post, we’ve reviewed various OAuth 2.0 grants, each catering to specific application needs, The authorization code grant ensures secure access for web applications (and offers additional security with the PKCE extension), and the client credentials grant is ideal for machine-to-machine authentication. Amazon Cognito acts as an encompassing identity platform, streamlining user authentication, authorization, and integration. By using these grants and the features provided by Cognito, developers can enhance security and the user experience in their applications. For more information and examples, see OAuth 2.0 grants in the Cognito Developer Guide.

Now that you understand implementing OAuth 2.0 grants in Amazon Cognito, see How to customize access tokens in Amazon Cognito user pools to learn about customizing access tokens to make fine-grained authorization decisions and provide a differentiated end-user experience.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Prashob Krishnan

Prashob Krishnan

Prashob is a Denver-based Technical Account Manager at AWS. Prashob is passionate about security. He enjoys working with customers to solve their technical challenges and help build a secure scalable architecture on the AWS Cloud.

Ishva Kanani

Ishva Kanani

Ishva is an Associate Security Consultant based in Fairfax, Virginia. She has a strong interest in security and enjoys assisting customers with securing cloud migrations and accelerating their journeys to the cloud and within the cloud ecosystem by delivering innovative solutions.

Keerthana Ganesh

Keerthana Ganesh

Keerthana is a Security and Compliance Solutions Architect at AWS based in the UK. She works with customers in EMEA and plays an advisory role on how to build secure architectures.

Improve healthcare services through patient 360: A zero-ETL approach to enable near real-time data analytics

Post Syndicated from Saeed Barghi original https://aws.amazon.com/blogs/big-data/improve-healthcare-services-through-patient-360-a-zero-etl-approach-to-enable-near-real-time-data-analytics/

Healthcare providers have an opportunity to improve the patient experience by collecting and analyzing broader and more diverse datasets. This includes patient medical history, allergies, immunizations, family disease history, and individuals’ lifestyle data such as workout habits. Having access to those datasets and forming a 360-degree view of patients allows healthcare providers such as claim analysts to see a broader context about each patient and personalize the care they provide for every individual. This is underpinned by building a complete patient profile that enables claim analysts to identify patterns, trends, potential gaps in care, and adherence to care plans. They can then use the result of their analysis to understand a patient’s health status, treatment history, and past or upcoming doctor consultations to make more informed decisions, streamline the claim management process, and improve operational outcomes. Achieving this will also improve general public health through better and more timely interventions, identify health risks through predictive analytics, and accelerate the research and development process.

AWS has invested in a zero-ETL (extract, transform, and load) future so that builders can focus more on creating value from data, instead of having to spend time preparing data for analysis. The solution proposed in this post follows a zero-ETL approach to data integration to facilitate near real-time analytics and deliver a more personalized patient experience. The solution uses AWS services such as AWS HealthLake, Amazon Redshift, Amazon Kinesis Data Streams, and AWS Lake Formation to build a 360 view of patients. These services enable you to collect and analyze data in near real time and put a comprehensive data governance framework in place that uses granular access control to secure sensitive data from unauthorized users.

Zero-ETL refers to a set of features on the AWS Cloud that enable integrating different data sources with Amazon Redshift:

Solution overview

Organizations in the healthcare industry are currently spending a significant amount of time and money on building complex ETL pipelines for data movement and integration. This means data will be replicated across multiple data stores via bespoke and in some cases hand-written ETL jobs, resulting in data inconsistency, latency, and potential security and privacy breaches.

With support for querying cross-account Apache Iceberg tables via Amazon Redshift, you can now build a more comprehensive patient-360 analysis by querying all patient data from one place. This means you can seamlessly combine information such as clinical data stored in HealthLake with data stored in operational databases such as a patient relationship management system, together with data produced from wearable devices in near real-time. Having access to all this data enables healthcare organizations to form a holistic view of patients, improve care coordination across multiple organizations, and provide highly personalized care for each individual.

The following diagram depicts the high-level solution we build to achieve these outcomes.

Deploy the solution

You can use the following AWS CloudFormation template to deploy the solution components:

This stack creates the following resources and necessary permissions to integrate the services:

AWS Solution setup

AWS HealthLake

AWS HealthLake enables organizations in the health industry to securely store, transform, transact, and analyze health data. It stores data in HL7 FHIR format, which is an interoperability standard designed for quick and efficient exchange of health data. When you create a HealthLake data store, a Fast Healthcare Interoperability Resources (FHIR) data repository is made available via a RESTful API endpoint. Simultaneously and as part of AWS HealthLake managed service, the nested JSON FHIR data undergoes an ETL process and is stored in Apache Iceberg open table format in Amazon S3.

To create an AWS HealthLake data store, refer to Getting started with AWS HealthLake. Make sure to select the option Preload sample data when creating your data store.

In real-world scenarios and when you use AWS HealthLake in production environments, you don’t need to load sample data into your AWS HealthLake data store. Instead, you can use FHIR REST API operations to manage and search resources in your AWS HealthLake data store.

We use two tables from the sample data stored in HealthLake: patient and allergyintolerance.

Query AWS HealthLake tables with Redshift Serverless

Amazon Redshift is the data warehousing service available on the AWS Cloud that provides up to six times better price-performance than any other cloud data warehouses in the market, with a fully managed, AI-powered, massively parallel processing (MPP) data warehouse built for performance, scale, and availability. With continuous innovations added to Amazon Redshift, it is now more than just a data warehouse. It enables organizations of different sizes and in different industries to access all the data they have in their AWS environments and analyze it from one single location with a set of features under the zero-ETL umbrella. Amazon Redshift integrates with AWS HealthLake and data lakes through Redshift Spectrum and Amazon S3 auto-copy features, enabling you to query data directly from files on Amazon S3.

Query AWS HealthLake data with Amazon Redshift

Amazon Redshift makes it straightforward to query the data stored in S3-based data lakes with automatic mounting of an AWS Glue Data Catalog in the Redshift query editor v2. This means you no longer have to create an external schema in Amazon Redshift to use the data lake tables cataloged in the Data Catalog. To get started with this feature, see Querying the AWS Glue Data Catalog. After it is set up and you’re connected to the Redshift query editor v2, complete the following steps:

  1. Validate that your tables are visible in the query editor V2. The Data Catalog objects are listed under the awsdatacatalog database.

FHIR data stored in AWS HealthLake is highly nested. To learn about how to un-nest semi-structured data with Amazon Redshift, see Tutorial: Querying nested data with Amazon Redshift Spectrum.

  1. Use the following query to un-nest the allergyintolerance and patient tables, join them together, and get patient details and their allergies:
    WITH patient_allergy AS 
    (
        SELECT
            resourcetype, 
            c AS allery_category,
            a."patient"."reference",
            SUBSTRING(a."patient"."reference", 9, LEN(a."patient"."reference")) AS patient_id,
            a.recordeddate AS allergy_record_date,
            NVL(cd."code", 'NA') AS allergy_code,
            NVL(cd.display, 'NA') AS allergy_description
    
        FROM "awsdatacatalog"."datastore_01_179674d36391d68926a8d74c12599306_healthlake_view"."allergyintolerance" a
                LEFT JOIN a.category c ON TRUE
                LEFT JOIN a.reaction r ON TRUE
                LEFT JOIN r.manifestation m ON TRUE
                LEFT JOIN m.coding cd ON TRUE
    ), patinet_info AS
    (
        SELECT id,
                gender,
                g as given_name,
                n.family as family_name,
                pr as prefix
    
        FROM "awsdatacatalog"."datastore_01_179674d36391d68926a8d74c12599306_healthlake_view"."patient" p
                LEFT JOIN p.name n ON TRUE
                LEFT JOIN n.given g ON TRUE
                LEFT JOIN n.prefix pr ON TRUE
    )
    SELECT DISTINCT p.id, 
            p.gender, 
            p.prefix,
            p.given_name,
            p.family_name,
            pa.allery_category,
            pa.allergy_code,
            pa.allergy_description
    from patient_allergy pa
        JOIN patinet_info p
            ON pa.patient_id = p.id
    ORDER BY p.id, pa.allergy_code
    ;
    

To eliminate the need for Amazon Redshift to un-nest data every time a query is run, you can create a materialized view to hold un-nested and flattened data. Materialized views are an effective mechanism to deal with complex and repeating queries. They contain a precomputed result set, based on a SQL query over one or more base tables. You can issue SELECT statements to query a materialized view, in the same way that you can query other tables or views in the database.

  1. Use the following SQL to create a materialized view. You use it later to build a complete view of patients:
    CREATE MATERIALIZED VIEW patient_allergy_info AUTO REFRESH YES AS
    WITH patient_allergy AS 
    (
        SELECT
            resourcetype, 
            c AS allery_category,
            a."patient"."reference",
            SUBSTRING(a."patient"."reference", 9, LEN(a."patient"."reference")) AS patient_id,
            a.recordeddate AS allergy_record_date,
            NVL(cd."code", 'NA') AS allergy_code,
            NVL(cd.display, 'NA') AS allergy_description
    
        FROM
            "awsdatacatalog"."datastore_01_179674d36391d68926a8d74c12599306_healthlake_view"."allergyintolerance" a
                LEFT JOIN a.category c ON TRUE
                LEFT JOIN a.reaction r ON TRUE
                LEFT JOIN r.manifestation m ON TRUE
                LEFT JOIN m.coding cd ON TRUE
    ), patinet_info AS
    (
        SELECT id,
                gender,
                g as given_name,
                n.family as family_name,
                pr as prefix
    
        FROM "awsdatacatalog"."datastore_01_179674d36391d68926a8d74c12599306_healthlake_view"."patient" p
                LEFT JOIN p.name n ON TRUE
                LEFT JOIN n.given g ON TRUE
                LEFT JOIN n.prefix pr ON TRUE
    )
    SELECT DISTINCT p.id, 
            p.gender, 
            p.prefix,
            p.given_name,
            p.family_name,
            pa.allery_category,
            pa.allergy_code,
            pa.allergy_description
    from patient_allergy pa
        JOIN patinet_info p
            ON pa.patient_id = p.id
    ORDER BY p.id, pa.allergy_code
    ;
    

You have confirmed you can query data in AWS HealthLake via Amazon Redshift. Next, you set up zero-ETL integration between Amazon Redshift and Amazon Aurora MySQL.

Set up zero-ETL integration between Amazon Aurora MySQL and Redshift Serverless

Applications such as front-desk software, which are used to schedule appointments and register new patients, store data in OLTP databases such as Aurora. To get data out of OLTP databases and have them ready for analytics use cases, data teams might have to spend a considerable amount of time to build, test, and deploy ETL jobs that are complex to maintain and scale.

With the Amazon Redshift zero-ETL integration with Amazon Aurora MySQL, you can run analytics on the data stored in OLTP databases and combine them with the rest of the data in Amazon Redshift and AWS HealthLake in near real time. In the next steps in this section, we connect to a MySQL database and set up zero-ETL integration with Amazon Redshift.

Connect to an Aurora MySQL database and set up data

Connect to your Aurora MySQL database using your editor of choice using AdminUsername and AdminPassword that you entered when running the CloudFormation stack. (For simplicity, it is the same for Amazon Redshift and Aurora.)

When you’re connected to your database, complete the following steps:

  1. Create a new database by running the following command:
    CREATE DATABASE front_desk_app_db;

  2. Create a new table. This table simulates storing patient information as they visit clinics and other healthcare centers. For simplicity and to demonstrate specific capabilities, we assume that patient IDs are the same in AWS HealthLake and the front-of-office application. In real-world scenarios, this can be a hashed version of a national health care number:
    CREATE TABLE patient_appointment ( 
          patient_id varchar(250), 
          gender varchar(1), 
          date_of_birth date, 
          appointment_datetime datetime, 
          phone_number varchar(15), 
          PRIMARY KEY (patient_id, appointment_datetime) 
    );

Having a primary key in the table is mandatory for zero-ETL integration to work.

  1. Insert new records into the source table in the Aurora MySQL database. To demonstrate the required functionalities, make sure the patient_id of the sample records inserted into the MySQL database match the ones in AWS HealthLake. Replace [patient_id_1] and [patient_id_2] in the following query with the ones from the Redshift query you ran previously (the query that joined allergyintolerance and patient):
    INSERT INTO front_desk_app_db.patient_appointment (patient_id, gender, date_of_birth, appointment_datetime, phone_number)
    
    VALUES([PATIENT_ID_1], 'F', '1988-7-04', '2023-12-19 10:15:00', '0401401401'),
    ([PATIENT_ID_1], 'F', '1988-7-04', '2023-09-19 11:00:00', '0401401401'),
    ([PATIENT_ID_1], 'F', '1988-7-04', '2023-06-06 14:30:00', '0401401401'),
    ([PATIENT_ID_2], 'F', '1972-11-14', '2023-12-19 08:15:00', '0401401402'),
    ([PATIENT_ID_2], 'F', '1972-11-14', '2023-01-09 12:15:00', '0401401402');

Now that your source table is populated with sample records, you can set up zero-ETL and have data ingested into Amazon Redshift.

Set up zero-ETL integration between Amazon Aurora MySQL and Amazon Redshift

Complete the following steps to create your zero-ETL integration:

  1. On the Amazon RDS console, choose Databases in the navigation pane.
  2. Choose the DB identifier of your cluster (not the instance).
  3. On the Zero-ETL Integration tab, choose Create zero-ETL integration.
  4. Follow the steps to create your integration.

Create a Redshift database from the integration

Next, you create a target database from the integration. You can do this by running a couple of simple SQL commands on Amazon Redshift. Log in to the query editor V2 and run the following commands:

  1. Get the integration ID of the zero-ETL you set up between your source database and Amazon Redshift:
    SELECT * FROM svv_integration;

  2. Create a database using the integration ID:
    CREATE DATABASE ztl_demo FROM INTEGRATION '[INTEGRATION_ID ';

  3. Query the database and validate that a new table is created and populated with data from your source MySQL database:
    SELECT * FROM ztl_demo.front_desk_app_db.patient_appointment;

It might take a few seconds for the first set of records to appear in Amazon Redshift.

This shows that the integration is working as expected. To validate it further, you can insert a new record in your Aurora MySQL database, and it will be available in Amazon Redshift for querying in near real time within a few seconds.

Set up streaming ingestion for Amazon Redshift

Another aspect of zero-ETL on AWS, for real-time and streaming data, is realized through Amazon Redshift Streaming Ingestion. It provides low-latency, high-speed ingestion of streaming data from Kinesis Data Streams and Amazon MSK. It lowers the effort required to have data ready for analytics workloads, lowers the cost of running such workloads on the cloud, and decreases the operational burden of maintaining the solution.

In the context of healthcare, understanding an individual’s exercise and movement patterns can help with overall health assessment and better treatment planning. In this section, you send simulated data from wearable devices to Kinesis Data Streams and integrate it with the rest of the data you already have access to from your Redshift Serverless data warehouse.

For step-by-step instructions, refer to Real-time analytics with Amazon Redshift streaming ingestion. Note the following steps when you set up streaming ingestion for Amazon Redshift:

  1. Select wearables_stream and use the following template when sending data to Amazon Kinesis Data Streams via Kinesis Data Generator, to simulate data generated by wearable devices. Replace [PATIENT_ID_1] and [PATIENT_ID_2] with the patient IDs you earlier when inserting new records into your Aurora MySQL table:
    {
       "patient_id": "{{random.arrayElement(["[PATIENT_ID_1]"," [PATIENT_ID_2]"])}}",
       "steps_increment": "{{random.arrayElement(
          [0,1]
       )}}",
       "heart_rate": {{random.number( 
          {
             "min":45,
             "max":120}
       )}}
    }

  2. Create an external schema called from_kds by running the following query and replacing [IAM_ROLE_ARN] with the ARN of the role created by the CloudFormation stack (Patient360BlogRole):
    CREATE EXTERNAL SCHEMA from_kds
    FROM KINESIS
    IAM_ROLE '[IAM_ROLE_ARN]';

  3. Use the following SQL when creating a materialized view to consume data from the stream:
    CREATE MATERIALIZED VIEW patient_wearable_data AUTO REFRESH YES AS 
    SELECT approximate_arrival_timestamp, 
          JSON_PARSE(kinesis_data) as Data FROM from_kds."wearables_stream" 
    WHERE CAN_JSON_PARSE(kinesis_data);

  4. To validate that streaming ingestion works as expected, refresh the materialized view to get the data you already sent to the data stream and query the table to make sure data has landed in Amazon Redshift:
    REFRESH MATERIALIZED VIEW patient_wearable_data;
    
    SELECT *
    FROM patient_wearable_data
    ORDER BY approximate_arrival_timestamp DESC;

Query and analyze patient wearable data

The results in the data column of the preceding query are in JSON format. Amazon Redshift makes it straightforward to work with semi-structured data in JSON format. It uses PartiQL language to offer SQL-compatible access to relational, semi-structured, and nested data. Use the following query to flatten data:

SELECT data."patient_id"::varchar AS patient_id,       
      data."steps_increment"::integer as steps_increment,       
      data."heart_rate"::integer as heart_rate, 
      approximate_arrival_timestamp 
FROM patient_wearable_data 
ORDER BY approximate_arrival_timestamp DESC;

The result looks like the following screenshot.

Now that you know how to flatten JSON data, you can analyze it further. Use the following query to get the number of minutes a patient has been physically active per day, based on their heart rate (greater than 80):

WITH patient_wearble_flattened AS
(
   SELECT data."patient_id"::varchar AS patient_id,
      data."steps_increment"::integer as steps_increment,
      data."heart_rate"::integer as heart_rate,
      approximate_arrival_timestamp,
      DATE(approximate_arrival_timestamp) AS date_received,
      extract(hour from approximate_arrival_timestamp) AS    hour_received,
      extract(minute from approximate_arrival_timestamp) AS minute_received
   FROM patient_wearable_data
), patient_active_minutes AS
(
   SELECT patient_id,
      date_received,
      hour_received,
      minute_received,
      avg(heart_rate) AS heart_rate
   FROM patient_wearble_flattened
   GROUP BY patient_id,
      date_received,
      hour_received,
      minute_received
   HAVING avg(heart_rate) > 80
)
SELECT patient_id,
      date_received,
      COUNT(heart_rate) AS active_minutes_count
FROM patient_active_minutes
GROUP BY patient_id,
      date_received
ORDER BY patient_id,
      date_received;

Create a complete patient 360

Now that you are able to query all patient data with Redshift Serverless, you can combine the three datasets you used in this post and form a comprehensive patient 360 view with the following query:

WITH patient_appointment_info AS
(
      SELECT "patient_id",
         "gender",
         "date_of_birth",
         "appointment_datetime",
         "phone_number"
      FROM ztl_demo.front_desk_app_db.patient_appointment
),
patient_wearble_flattened AS
(
      SELECT data."patient_id"::varchar AS patient_id,
         data."steps_increment"::integer as steps_increment,
         data."heart_rate"::integer as heart_rate,
         approximate_arrival_timestamp,
         DATE(approximate_arrival_timestamp) AS date_received,
         extract(hour from approximate_arrival_timestamp) AS hour_received,
         extract(minute from approximate_arrival_timestamp) AS minute_received
      FROM patient_wearable_data
), patient_active_minutes AS
(
      SELECT patient_id,
         date_received,
         hour_received,
         minute_received,
         avg(heart_rate) AS heart_rate
      FROM patient_wearble_flattened
      GROUP BY patient_id,
         date_received,
         hour_received,
         minute_received
         HAVING avg(heart_rate) > 80
), patient_active_minutes_count AS
(
      SELECT patient_id,
         date_received,
         COUNT(heart_rate) AS active_minutes_count
      FROM patient_active_minutes
      GROUP BY patient_id,
         date_received
)
SELECT pai.patient_id,
      pai.gender,
      pai.prefix,
      pai.given_name,
      pai.family_name,
      pai.allery_category,
      pai.allergy_code,
      pai.allergy_description,
      ppi.date_of_birth,
      ppi.appointment_datetime,
      ppi.phone_number,
      pamc.date_received,
      pamc.active_minutes_count
FROM patient_allergy_info pai
      LEFT JOIN patient_active_minutes_count pamc
            ON pai.patient_id = pamc.patient_id
      LEFT JOIN patient_appointment_info ppi
            ON pai.patient_id = ppi.patient_id
GROUP BY pai.patient_id,
      pai.gender,
      pai.prefix,
      pai.given_name,
      pai.family_name,
      pai.allery_category,
      pai.allergy_code,
      pai.allergy_description,
      ppi.date_of_birth,
      ppi.appointment_datetime,
      ppi.phone_number,
      pamc.date_received,
      pamc.active_minutes_count
ORDER BY pai.patient_id,
      pai.gender,
      pai.prefix,
      pai.given_name,
      pai.family_name,
      pai.allery_category,
      pai.allergy_code,
      pai.allergy_description,
      ppi.date_of_birth DESC,
      ppi.appointment_datetime DESC,
      ppi.phone_number DESC,
      pamc.date_received,
      pamc.active_minutes_count

You can use the solution and queries used here to expand the datasets used in your analysis. For example, you can include other tables from AWS HealthLake as needed.

Clean up

To clean up resources you created, complete the following steps:

  1. Delete the zero-ETL integration between Amazon RDS and Amazon Redshift.
  2. Delete the CloudFormation stack.
  3. Delete AWS HealthLake data store

Conclusion

Forming a comprehensive 360 view of patients by integrating data from various different sources offers numerous benefits for organizations operating in the healthcare industry. It enables healthcare providers to gain a holistic understanding of a patient’s medical journey, enhances clinical decision-making, and allows for more accurate diagnosis and tailored treatment plans. With zero-ETL features for data integration on AWS, it is effortless to build a view of patients securely, cost-effectively, and with minimal effort.

You can then use visualization tools such as Amazon QuickSight to build dashboards or use Amazon Redshift ML to enable data analysts and database developers to train machine learning (ML) models with the data integrated through Amazon Redshift zero-ETL. The result is a set of ML models that are trained with a broader view into patients, their medical history, and their lifestyle, and therefore enable you make more accurate predictions about their upcoming health needs.


About the Authors

Saeed Barghi is a Sr. Analytics Specialist Solutions Architect specializing in architecting enterprise data platforms. He has extensive experience in the fields of data warehousing, data engineering, data lakes, and AI/ML. Based in Melbourne, Australia, Saeed works with public sector customers in Australia and New Zealand.

Satesh Sonti is a Sr. Analytics Specialist Solutions Architect based out of Atlanta, specialized in building enterprise data platforms, data warehousing, and analytics solutions. He has over 17 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Successfully conduct a proof of concept in Amazon Redshift

Post Syndicated from Ziad Wali original https://aws.amazon.com/blogs/big-data/successfully-conduct-a-proof-of-concept-in-amazon-redshift/

Amazon Redshift is a fast, scalable, and fully managed cloud data warehouse that allows you to process and run your complex SQL analytics workloads on structured and semi-structured data. It also helps you securely access your data in operational databases, data lakes, or third-party datasets with minimal movement or copying of data. Tens of thousands of customers use Amazon Redshift to process large amounts of data, modernize their data analytics workloads, and provide insights for their business users.

In this post, we discuss how to successfully conduct a proof of concept in Amazon Redshift by going through the main stages of the process, available tools that accelerate implementation, and common use cases.

Proof of concept overview

A proof of concept (POC) is a process that uses representative data to validate whether a technology or service fulfills a customer’s technical and business requirements. By testing the solution against key metrics, a POC provides insights that allow you to make an informed decision on the suitability of the technology for the intended use case.

There are three major POC validation areas:

  • Workload – Take a representative portion of an existing workload and test it on Amazon Redshift, such as an extract, transform, and load (ETL) process, reporting, or management
  • Capability – Demonstrate how a specific Amazon Redshift feature, such as zero-ETL integration with Amazon Redshift, data sharing, or Amazon Redshift Spectrum, can simplify or enhance your overall architecture
  • Architecture – Understand how Amazon Redshift fits into a new or existing architecture along with other AWS services and tools

A POC is not:

  • Planning and implementing a large-scale migration
  • User-facing deployments, such as deploying a configuration for user testing and validation over extended periods (this is more of a pilot)
  • End-to-end implementation of a use case (this is more of a prototype)

Proof of concept process

For a POC to be successful, it is recommended to follow and apply a well-defined and structured process. For a POC on Amazon Redshift, we recommend a three-phase process of discovery, implementation, and evaluation.

Discovery phase

The discovery phase is considered the most essential among the three phases and the longest. It defines through multiple sessions the scope of the POC and the list of tasks that need to be completed and later evaluated. The scope should contain inputs and data points on the current architecture as well as the target architecture. The following items need to be defined and documented to have a defined scope for the POC:

  • Current state architecture and its challenges
  • Business goals and the success criteria of the POC (such as cost, performance, and security) along with their associated priorities
  • Evaluation criteria that will be used to evaluate and interpret the success criteria, such as service-level agreements (SLAs)
  • Target architecture (the communication between the services and tools that will be used during the implementation of the POC)
  • Dataset and the list of tables and schemas

After the scope has been clearly defined, you should proceed with defining and planning the list of tasks that need to be run during the next phase in order to implement the scope. Also, depending on the technical familiarity with the latest developments in Amazon Redshift, a technical enablement session on Amazon Redshift is also highly recommended before starting the implementation phase.

Optionally, a responsibility assignment matrix (RAM) is recommended, especially in large POCs.

Implementation phase

The implementation phase takes the output of the previous phase as input. It consists of the following steps:

  1. Set up the environment by respecting the defined POC architecture.
  2. Complete the implementation tasks such as data ingestion and performance testing.
  3. Collect data metrics and statistics on the completed tasks.
  4. Analyze the data and then optimize as necessary.

Evaluation phase

The evaluation phase is the POC assessment and the final step of the process. It aggregates the implementation results of the preceding phase, interprets them, and evaluates the success criteria described in the discovery phase.

It is recommended to use percentiles instead of averages whenever possible for a better interpretation.

Challenges

In this section, we discuss the major challenges that you may encounter while planning your POC.

Scope

You may face challenges during the discovery phase while defining the scope of the POC, especially in complex environments. You should focus on the crucial requirements and prioritized success criteria that need to be evaluated so you avoid ending up with a small migration project instead of a POC. In terms of technical content (such as data structures, transformation jobs, and reporting queries), make sure to identify and consider as little as possible of the content that will still provide you with all the necessary information at the end of the implementation phase in order to assess the defined success criteria. Additionally, document any assumptions you are making.

Time

A time period should be defined for any POC project to ensure it stays focused and achieves clear results. Without an established time frame, scope creep can occur as requirements shift and unnecessary features get added. This may lead to misleading evaluations about the technology or concept being tested. The duration set for the POC depends on factors like workload complexity and resource availability. If a period such as 3 weeks has been committed to already without accounting for these considerations, the scope and planned content should be scaled to feasibly fit that fixed time period.

Cost

Cloud services operate on a pay-as-you-go model, and estimating costs accurately can be challenging during a POC. Overspending or underestimating resource requirements can impact budget allocations. It’s important to carefully estimate the initial sizing of the Redshift cluster, monitor resource usage closely, and consider setting service limits along with AWS Budget alerts to avoid unexpected expenditures.

Technical

The team running the POC has to be ready for initial technical challenges, especially during environment setup, data ingestion, and performance testing. Each data warehouse technology has its own design and architecture, which sometimes requires some initial tuning at the data structure or query level. This is an expected challenge that needs to be considered in the implementation phase timeline. Having a technical enablement session beforehand can alleviate such hurdles.

Amazon Redshift POC tools and features

In this section, we discuss tools that you can adapt based on the specific requirements and nature of the POC being conducted. It’s essential to choose tools that align with the scope and technologies involved.

AWS Analytics Automation Toolkit

The AWS Analytics Automation Toolkit enables automatic provisioning and integration of not only Amazon Redshift, but database migration services like AWS Database Migration Service (AWS DMS), AWS Schema Conversion Tool (AWS SCT), and Apache JMeter. This toolkit is essential in most POCs because it automates the provisioning of infrastructure and setup of the necessary environment.

AWS SCT

The AWS SCT makes heterogeneous database migrations predictable, secure, and fast by automatically converting the majority of the database code and storage objects to a format that is compatible with the target database. Any objects that can’t be automatically converted are clearly marked so that they can be manually converted to complete the migration.

In the context of a POC, the AWS SCT becomes crucial by streamlining and enhancing the efficiency of the schema conversion process from one database system to another. Given the time-sensitive nature of POCs, the AWS SCT automates the conversion process, facilitating planning, and estimation of time and efforts. Additionally, the AWS SCT plays a role in identifying potential compatibility issues, data mapping challenges, or other hurdles at an early stage of the process.

Furthermore, the database migration assessment report summarizes all the action items for schemas that can’t be converted automatically to your target database. Getting started with AWS SCT is a straightforward process. Also, consider following the best practices for AWS SCT.

Amazon Redshift auto-copy

The Amazon Redshift auto-copy (preview) feature can automate data ingestion from Amazon Simple Storage Service (Amazon S3) to Amazon Redshift with a simple SQL command. COPY statements are invoked and start loading data when Amazon Redshift auto-copy detects new files in the specified S3 prefixes. This also makes sure that end-users have the latest data available in Amazon Redshift shortly after the source files are available.

You can use this feature for the purpose of data ingestion throughout the POC. To learn more about ingesting from files located in Amazon S3 using a SQL command, refer to Simplify data ingestion from Amazon S3 to Amazon Redshift using auto-copy (preview). The post also shows you how to enable auto-copy using COPY jobs, how to monitor jobs, and considerations and best practices.

Redshift Auto Loader

The custom Redshift Auto Loader framework automatically creates schemas and tables in the target database and continuously loads data from Amazon S3 to Amazon Redshift. You can use this during the data ingestion phase of the POC. Deploying and setting up the Redshift Auto Loader framework to transfer files from Amazon S3 to Amazon Redshift is a straightforward process.

For more information, refer to Migrate from Google BigQuery to Amazon Redshift using AWS Glue and Custom Auto Loader Framework.

Apache JMeter

Apache JMeter is an open-source load testing application written in Java that you can use to load test web applications, backend server applications, databases, and more. In a database context, it’s an extremely valuable tool for repeating benchmark tests in a consistent manner, simulating concurrency workloads, and scalability testing on different database configurations.

When implementing your POC, benchmarking Amazon Redshift is often one of the main components of evaluation and a key source of insight into the price-performance of different Amazon Redshift configurations. With Apache JMeter, you can construct high-quality benchmark tests for Amazon Redshift.

Workload Replicator

If you are currently using Amazon Redshift and looking to replicate your existing production workload or isolate specific workloads in a POC, you can use the Workload Replicator to run them across different configurations of Redshift clusters (ra3.xlplus, ra3.4xl,ra3.16xl, serverless) for performance evaluation and comparison.

This utility has the ability to mimic COPY and UNLOAD workloads and can run the transactions and queries in the same time interval as they’re run in the production cluster. However, it’s crucial to assess the limitations of the utility and AWS Identity and Access Management (IAM) security and compliance requirements.

Node Configuration Comparison utility

If you’re using Amazon Redshift and have stringent SLAs for query performance in your Amazon Redshift cluster, or you want to explore different Amazon Redshift configurations based on the price-performance of your workload, you can use the Amazon Redshift Node Configuration Comparison utility.

This utility helps evaluate performance of your queries using different Redshift cluster configurations in parallel and compares the end results to find the best cluster configuration that meets your need. Similarly, If you’re already using Amazon Redshift and want to migrate from your existing DC2 or DS2 instances to RA3, you can refer to our recommendations on node count and type when upgrading. Before doing that, you can use this utility in your POC to evaluate the new cluster’s performance by replaying your past workloads, which integrates with the Workload Replicator utility to evaluate performance metrics for different Amazon Redshift configurations to meet your needs.

This utility functions in a fully automated manner and has similar limitations as the workload replicator. However, it requires full permissions across various services for the user running the AWS CloudFormation stack.

Use cases

You have the opportunity to explore various functionalities and aspects of Amazon Redshift by defining and selecting a business use case you want to validate during the POC. In this section, we discuss some specific use cases you can explore using a POC.

Functionality evaluation

Amazon Redshift consists of a set of functionalities and options that simplify data pipelines and effortlessly integrate with other services. You can use a POC to test and evaluate one or more of those capabilities before refactoring your data pipeline and implementing them in your ecosystem. Functionalities could be existing features or new ones such as zero-ETL integration, streaming ingestion, federated queries, or machine learning.

Workload isolation

You can use the data sharing feature of Amazon Redshift to achieve workload isolation across diverse analytics use cases and achieve business-critical SLAs without duplicating or moving the data.

Amazon Redshift data sharing enables a producer cluster to share data objects with one or more consumer clusters, thereby eliminating data duplication. This facilitates collaboration across isolated clusters, allowing data to be shared for innovation and analytic services. Sharing can occur at various levels such as databases, schemas, tables, views, columns, and user-defined functions, offering fine-grained access control. It is recommended to use Workload Replicator for performance evaluation and comparison in a workload isolation POC.

The following sample architectures explain workload isolation using data sharing. The first diagram illustrates the architecture before using data sharing.

The following diagram illustrates the architecture with data sharing.

Migrating to Amazon Redshift

If you’re interested in migrating from your existing data warehouse platform to Amazon Redshift, you can try out Amazon Redshift by developing a POC on a selected business use case. In this type of POC, it is recommended to use the AWS Analytics Automation Toolkit for setting up the environment, auto-copy or Redshift Auto Loader for data ingestion, and AWS SCT for schema conversion. When the development is complete, you can perform performance testing using Apache JMeter, which provides data points to measure price-performance and compare results with your existing platform. The following diagram illustrates this process.

Moving to Amazon Redshift Serverless

You can migrate your unpredictable and variable workloads to Amazon Redshift Serverless, which enables you to scale as and when needed and pay as per usage, making your infrastructure scalable and cost-efficient. If you’re migrating your full workload from provisioned (DC2, RA3) to serverless, you can use the Node Configuration Comparison utility for performance evaluation. The following diagram illustrates this workflow.

Conclusion

In a competitive environment, conducting a successful proof of concept is a strategic imperative for businesses aiming to validate the feasibility and effectiveness of new solutions. Amazon Redshift provides you with better price-performance compared to other cloud-centered data warehouses, and a large list of features that help you modernize and optimize your data pipelines. For more details, see Amazon Redshift continues its price-performance leadership.

With the process discussed in this post and by choosing the tools needed for your specific use case, you can accelerate the process of conducting a POC. This allows you to collect the data metrics that can help you understand the potential challenges, benefits, and implications of implementing the proposed solution on a larger scale. A POC provides essential data points that evaluate price-performance as well as feasibility, which plays a vital role in decision-making.


About the Authors

Ziad WALI is an Acceleration Lab Solutions Architect at Amazon Web Services. He has over 10 years of experience in databases and data warehousing, where he enjoys building reliable, scalable, and efficient solutions. Outside of work, he enjoys sports and spending time in nature.

Omama Khurshid is an Acceleration Lab Solutions Architect at Amazon Web Services. She focuses on helping customers across various industries build reliable, scalable, and efficient solutions. Outside of work, she enjoys spending time with her family, watching movies, listening to music, and learning new technologies.

Srikant Das is an Acceleration Lab Solutions Architect at Amazon Web Services. His expertise lies in constructing robust, scalable, and efficient solutions. Beyond the professional sphere, he finds joy in travel and shares his experiences through insightful blogging on social media platforms.

The PostgreSQL community mourns Simon Riggs

Post Syndicated from corbet original https://lwn.net/Articles/966868/

The PostgreSQL community is dealing with the loss of Simon Riggs, who
passed away on March 26:

Simon was responsible for many of the enterprise features we find
in PostgreSQL today, including point in time recovery, hot standby,
and synchronous replication. He was the founder of 2ndQuadrant
which employed many of the PostgreSQL developers, later becoming
part of EDB where he worked as a Postgres Fellow until his
retirement. He was responsible for the UK PostgreSQL conferences
for many years until he passed that responsibility to PostgreSQL
Europe last year.

[$] High-performance computing with Ubuntu

Post Syndicated from daroc original https://lwn.net/Articles/965516/

Jason Nucciarone and Felipe Reyes gave back-to-back talks
about high-performance computing (HPC) using Ubuntu at
SCALE this
year. Nucciarone talked about ongoing work packaging

Open OnDemand
— a web-based HPC cluster interface —
to make high-performance-computing clusters
more user friendly. Reyes presented on using
OpenStack — a cloud-computing platform
— to pass the performance benefits of one’s hardware through
to virtual machines (VMs) running on a cluster.

Security updates for Wednesday

Post Syndicated from jzb original https://lwn.net/Articles/966835/

Security updates have been issued by Debian (composer and nodejs), Fedora (w3m), Mageia (tomcat), Oracle (expat, firefox, go-toolset:ol8, grafana, grafana-pcp, nodejs:18, and thunderbird), Red Hat (dnsmasq, expat, kernel, kernel-rt, libreoffice, and squid), and SUSE (firefox, krb5, libvirt, and shadow).

Security Vulnerability in Saflok’s RFID-Based Keycard Locks

Post Syndicated from Bruce Schneier original https://www.schneier.com/blog/archives/2024/03/security-vulnerability-in-safloks-rfid-based-keycard-locks.html

It’s pretty devastating:

Today, Ian Carroll, Lennert Wouters, and a team of other security researchers are revealing a hotel keycard hacking technique they call Unsaflok. The technique is a collection of security vulnerabilities that would allow a hacker to almost instantly open several models of Saflok-brand RFID-based keycard locks sold by the Swiss lock maker Dormakaba. The Saflok systems are installed on 3 million doors worldwide, inside 13,000 properties in 131 countries. By exploiting weaknesses in both Dormakaba’s encryption and the underlying RFID system Dormakaba uses, known as MIFARE Classic, Carroll and Wouters have demonstrated just how easily they can open a Saflok keycard lock. Their technique starts with obtaining any keycard from a target hotel—say, by booking a room there or grabbing a keycard out of a box of used ones—then reading a certain code from that card with a $300 RFID read-write device, and finally writing two keycards of their own. When they merely tap those two cards on a lock, the first rewrites a certain piece of the lock’s data, and the second opens it.

Dormakaba says that it’s been working since early last year to make hotels that use Saflok aware of their security flaws and to help them fix or replace the vulnerable locks. For many of the Saflok systems sold in the last eight years, there’s no hardware replacement necessary for each individual lock. Instead, hotels will only need to update or replace the front desk management system and have a technician carry out a relatively quick reprogramming of each lock, door by door. Wouters and Carroll say they were nonetheless told by Dormakaba that, as of this month, only 36 percent of installed Safloks have been updated. Given that the locks aren’t connected to the internet and some older locks will still need a hardware upgrade, they say the full fix will still likely take months longer to roll out, at the very least. Some older installations may take years.

If ever. My guess is that for many locks, this is a permanent vulnerability.

Supporting Computing in England through our renewed partnership with Oak National Academy

Post Syndicated from Rik Cross original https://www.raspberrypi.org/blog/supporting-computing-in-england-with-oak-national-academy/

We are pleased to announce that we are renewing our partnership with Oak National Academy in England to provide an updated high-quality Computing curriculum and lesson materials for Key Stages 1 to 4.

In a computing classroom, a girl looks at a computer screen.

New curriculum and materials for the classroom

In 2021 we partnered with Oak National Academy to offer content for schools in England that supported young people to learn Computing at home while schools were closed as a result of the coronavirus pandemic.

A teacher and learner at a laptop doing coding.

In our renewed partnership, we will create new and updated materials for primary and secondary teachers to use in the classroom. These classroom units will be available for free on the Oak platform and will include everything a teacher needs to deliver engaging lessons, including slide decks, worksheets, quizzes, and accompanying videos for over 550 lessons. The units will cover both the general national Computing curriculum and the Computer Science GCSE, supporting teachers to provide a high-quality Computing offering to all students aged 5 to 16.

Secondary school age learners in a computing classroom.

These new resources will update the very successful Computing Curriculum and will be rigorously tested by a Computing subject expert group.

“I am delighted that we are continuing our partnership with Oak National Academy to support all teachers in England with world-leading resources for teaching Computing and Computer Science. This means that all teachers in England will have access to free, rigorous and tested classroom resources that they can adapt to suit their context and students.” – Philip Colligan, CEO

All our materials on the Oak platform will be free and openly available, and can be accessed by educators worldwide.

Research-informed, time-saving, and adaptable resources

As we did with The Computing Curriculum, we’ll design these teaching resources to model best practice, and they will be informed by leading research into pedagogy and computing education, as well as by user testing and feedback. 

Young learners at computers in a classroom.

The materials will bring teachers the added benefit of saving valuable time, and schools can choose to adapt and use the resources in the way that works best for their students

Supporting schools in England and worldwide

We have already started work and will begin releasing units of lessons in autumn 2024. All units across Key Stages 1 to 4 will be available by autumn 2025.

A teenager learning computer science.

We’re excited to continue our partnership with Oak National Academy to provide support to teachers and students in England. 

And as always, our comprehensive classroom resources can be downloaded for free, by anyone in the world, from our website.

The post Supporting Computing in England through our renewed partnership with Oak National Academy appeared first on Raspberry Pi Foundation.

The collective thoughts of the interwebz

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close