Wizards Unite Recaps Entire Story Arc
Posted on
by Theo Dwyer|
Since the end of J. K. Rowling‘s iconic Harry Potter series with the seventh novel, Harry Potter & the Deathly Hallows, the author’s Wizarding World has continued in multiple different ways. Rowling has penned her own spinoffs and direct continuations, including the popular Fantastic Beasts franchise where she serves as screenwriter and the controversial but successful Harry Potter & the Cursed Child, a theatrical play that Rowling plotted. What many readers may not realize, though, is that there is a monthly narrative that directly continues the adventures of the core characters, including Harry Potter, Hermine Granger, and Ron Weasley (who has had a particularly hard time of late). Niantic Labs’ mobile game, Harry Potter: Wizards Unite, is misunderstood by many to be purely a collection game due to a confusing launch in 2019. The game has developed into a task-driven quest to unlock pieces of narrative, working directly with Harry and Hermione to solve a mystery that will reveal the biggest threat to the Wizarding World since Voldemort: a cabal of dark witches and wizards called The Unforgivable.
Harry Potter: Wizards Unite Brilliant Event registry. Credit: Niantic
In an effort to catch readers and potential players up to their story, Niantic posted a summary of Harry Potter: Wizards Unite‘s first two years over on their official blog. You can preview the first portion of it right here:
Three years prior to the events of the game, five witches and wizards known collectively as the «London Five» vanished without a trace. The victims were Riya Patel, an Auror; Kit Gerrard, a reporter for the Daily Prophet; Timothy Hale, a Records Officer in the Ministry of Magic; Genevieve Blatt, the Head of the Department of Magical Artefacts; and Penelope Fawley, an Auror and wife of Unspeakable Grim Fawley.
A thorough investigation was conducted, but the case was ultimately closed due to budgetary reasons and a lack of results.
Grim Fawley, desperate to find his wife, pursued both mundane and magical means to bring her back home. This included an ancient spell rumoured to find what one desires most.
Grim disappeared under mysterious circumstances around the same time as the Calamity began.
The British Ministry of Magic and International Confederation of Wizards created the Statute of Secrecy (SOS) Task Force. Under the guidance of Constance Pickering and Hermione Granger, volunteers (like you!) were enlisted to combat the Calamity and prevent Statute of Secrecy breaches.
The Calamity started to experience «surges» – essentially spurts of heightened activity – of related Foundables that seemed to imply the magic was being guided.
Personally, I think that Harry Potter: Wizards Unite is a great game with a compelling story that creates within it a sense of urgency and excitement to get to the next chapter that the films just can’t create because of their scope and release schedule.
With the story now laid out at the above link and a major revelation coming in this week’s Brilliant Event, there has never been a more exciting time to play Harry Potter: Wizards Unite.
Posted in: Games, Harry Potter: Wizards Unite, Mobile Games, Niantic | Tagged: harry potter, Harry Potter: Wizards Unite, hpwu, Niantic, Wizards Unite
Enjoyed this? Please share on social media!
Dragon Ball Super Previews Power Absorbed: Dende & Hercule
Dragon Ball Super Card Game’s Power Absorbed gives the spotlights to Z-Warrior legends like Goku & Vegeta as well as supporters like Dende.
Mega Aerodactyl Raid Guide For Pokémon GO Players: January 2023
Mega Aerodactyl Raid Guide for Pokémon GO Players: Prepare yourself with a team of Rock-, Steel-, and Ice-types to defeat this raid.
The Cards Of Pokémon TCG: Silver Tempest Part 51: Gardevoir CR
Our spotlight on the Trainer Gallery subset of Pokémon TCG: Sword & Shield — Silver Tempest continues with Gardevoir, Jynx, and Malamar.
Tonight Is Tynamo Spotlight Hour In Pokémon GO: January 2023
Tonight is Tynamo Spotlight Hour in Pokémon GO. Can it be Shiny? What is the Spotlight Hour bonus? Prepare for the event right here.
Dragon Ball Super CG Value Watch: Fighter’s Ambition In January 2023
Dragon Ball Super CG Value Watch: Fighter’s Ambition in January 2023 notes a huge rise for an unlikely Secret Rare featuring a new character.
Pokémon TCG Japan: Scarlet & Violet Ex Preview: Wattrel Line
Pokémon TCG Japan: Scarlet ex, the first set of the Scarlet & Violet era, features the Electric/Flying-type Paldean bird, Wattrel.
Dragon Ball Super Previews Power Absorbed: Dende & Hercule
Dragon Ball Super Card Game’s Power Absorbed gives the spotlights to Z-Warrior legends like Goku & Vegeta as well as supporters like Dende.
Niantic Officially Launches NBA All-World This Week
Play in the NBA the Niantic way, as the mobile publisher has released NBA All-World for mobile devices today.
Mega Aerodactyl Raid Guide For Pokémon GO Players: January 2023
Mega Aerodactyl Raid Guide for Pokémon GO Players: Prepare yourself with a team of Rock-, Steel-, and Ice-types to defeat this raid.
The Last Starship Is Coming To Steam Next Fest In February
The Last Starship will be a part of Steam Next Fest in February, but before that, a demo will be made available to try out.
Star Trek Online: Refractions Launches For PC Today
Explore Star Trek Online: Refractions as the latest expansion brings Beverly Crusher into the frame of the Mirror Universe.
The Cards Of Pokémon TCG: Silver Tempest Part 51: Gardevoir CR
Our spotlight on the Trainer Gallery subset of Pokémon TCG: Sword & Shield — Silver Tempest continues with Gardevoir, Jynx, and Malamar.
The Complete Harry Potter Story
By
Tanner Fox
We break down the story for Harry Potter: Wizards Unite, Niantic’s follow-up to 2016’s augmented reality smash hit Pokémon Go.
Harry Potter Wizards Unite, Niantic’s pseudo-unexpected followup to 2016’s augmented reality smash hit Pokémon Go, debuted on the 21st of June. Though the launch was somewhat muddled, the game is still apparently on track to make around $10 million within the first month. The Harry Potter franchise is said to be worth nearly $25 billion, and the new mobile title is just one example of the multimedia license’s continued pop-culture prominence.
RELATED: 10 Pro Tips For Harry Potter Wizards Unite
That said, Wizards Unite has across the board been unfavorably compared to Pokémon Go, and the criticism is, in some cases, justified.
While PoGo was a form of wish fulfillment for wannabe Pokémon trainers the world over, nobody really asked for a weird, slightly tortured rip-off set in the wizarding world of everyone’s favorite cupboard-dwelling boy who lived. If you wanted to describe this game to an unaware friend in three words, it could quite literally be summed up as just, “Harry Potter Pokémon.”
via: harrypotterwizardsunite.com
The gameplay seems eager to contrive reasons for its existence, and, while there is indeed some sort of narrative to be found in Wizards Unite, the game isn’t overly-eager to present any of these elements beyond the initial set up. The central concept revolves around a near-apocalyptic event in the Harry Potter universe referred to as, “The Calamity.” This happening caused a ton of magical objects to be released into the world, and the player steps into the role of a witch or wizard tasked with capturing these rogue items before they expose the magical world to all of muggle-kind.
Magical folk are, for some reason never thoroughly detailed in any Harry Potter material, sworn to secrecy, though they never seem to do a great job of keeping quiet.
All of this will be explained in the game’s opening few moments, of course, but, for those who are interested in the finer details, there are a few story elements at play which aren’t directly divulged in the game itself. At this point, we need to give a huge shout out to YouTuber The Sylph, who combed through a ton of the game’s marketing material and extraneous lore fragments to piece together the scattered details of the plot.
via: themill.com
The crux of the story revolves around a mysterious wizard by the name of Grim Fowly—and, yes, this time it doesn’t seem to have been Voldemort’s doing. In fact, Harry Potter Wizards Unite takes place after the events of the books and movies, and actually features a much older version of the series’ primary protagonist. Fowly is allegedly responsible for The Calamity, and he’s been associated with a gang conducting magical malfeasance known as the London Five.
RELATED: Harry Potter Wizards Unite Won’t Last Like Pokémon Go
Fowly has connections to the Ministry of Magic, which, at this time, is headed by none other than Hermione Granger. For whatever reason, there’s a division of wizards and witches working within the Ministry dedicated to experiments and discoveries surrounding the concept of love. They have their own laboratory of sorts, which is referred to as the Love Room.
via: pottermore.com
Grim Fowly is known to have been entering and exiting the Love Room at strange times, and it’s implied that he’s been up to some pretty nefarious stuff. Actually, ‘nefarious’ may not exactly be the correct word, as some correspondence between himself and the London Five seems to suggest that he was experimenting on some sort of potion or spell which would somehow reunite him with his wife, Penelope. It’s implied that Penelope was a member of the Five and that she’s gone missing for reasons unknown or unexplained.
So, in an effort to find his missing wife, Fowly conducts these experiments and eventually starts some sort of reaction which sets off this whole sordid chain of events.
There are a few more details than that, but that’s the main gist of the game’s plot. No, it’s not exactly straightforward, but, then again, neither is anything in the Harry Potter universe.
via: harrypotterwizardsunite.com
Yet, how does this explain the ubiquitous presence of nearly every major character from the main novels and films? Well, it doesn’t really, and most of these people feel shoehorned in for the sake of brand recognition. In a way, it sort of damages the alternate reality experience. While a more toned-down experience would have helped users to ingratiate themselves in the magical world, the influx of familiar faces makes the game feel like just another tiresome, run-of-the-mill piece of media.
At the end of the day, the narrative shouldn’t really get in the way of anyone’s enjoyment of this experience. It may not be the most fully-fledged or well thought out mobile title, but, hey, Pokémon Go didn’t need to advance some sort of complicated narrative to attract players.
Quite honestly, all anyone really needs to know is that there are magical items all over the place, and you need to clean them up, lest the wizarding world be exposed. Fighting a hippogriff may not be as thrilling as catching a Pikachu, but it can still be a good time for those willing to look past some of the strange, contrived bits of narrative.
NEXT: First Harry Potter Wizards Unite Fan Festival To Be Held In US This Fall
Wizard Merlin — Knife
In this chapter, we will continue our journey through the Arthurian world. It would seem that everyone knows the main wizard of the cycle, but in fact he appeared in mythology as a completely independent figure, outside the legend of King Arthur. Merlin’s true origin is hidden in the depths of the Celtic lore, which for centuries has been told and rethought in Scotland and in the north of Britain, in Ulster and Wales. In this story, he appears not as a sage, but as a savage, not as a magician, but as a prophet. nine0003
Merlin is one of the most enduringly popular heroes of the Arthurian cycle, more mysterious than Arthur himself, and therefore exciting the imagination of many generations of authors.
One of the latest rethinking of his image is found in the 2008-2012 fantasy series of the same name, which tells about the wizard’s youth. The plot itself is not new: already in medieval texts, Merlin appeared more than once in the form of a child or a young man. In fact, as we shall see later, one of his main prototypes was a clairvoyant boy named Ambrose (or Ambrosius) from an early medieval legend. The novelty of the modern version was that the authors removed the age difference between Merlin and Arthur and made them peers, abandoning the classic model of the old mentor and the young king. nine0003
An earlier example of an original interpretation of the character can be found in T. H. White’s The Sword in the Stone (1938), which became the first part of a famous trilogy. In it, the wizard Merlin trains the future King Arthur, turning him into various creatures and sending him on an adventure. The novel gives a brilliant explanation of Merlin’s wisdom and insight: he has the strange ability to live in reverse time, thus he knows Arthur’s future from his past.
In paying homage to White, I will also talk about how the image of Merlin changed, in reverse order — from the latest representations to the earliest. nine0003
Let’s start with Geoffrey of Monmouth — this author was discussed in the previous chapter — who worked in the first half of the 12th century. Anyone who wants to get acquainted with the legendary history of Britain will have to deal with Geoffrey. In this book, we will meet with him more than once. Probably Geoffrey was a Briton: perhaps a Welsh or a Breton. It is obvious that he wrote for an Anglo-Norman audience, and his Welsh language, apparently, was mediocre. It was thanks to his «History of the Kings of Britain» that in medieval Europe, from about 1138, the fashion for King Arthur went. The Arthurian section of the book was only part of a project to create an extensive and coherent history of the island of Britain. It seems that the Normans, having conquered England, showed considerable interest in the legendary past of their new possessions, perhaps to overcome a sense of alienation.
nine0003
However, Galfrid’s first work written in Latin was not about Arthur, but about Merlin. In a strange text called «The Prophecies of Merlin» (Prophetiae Merlini), the wizard appears as a child, a sort of child prodigy. He spouts a series of deliberately confusing political prophecies, culminating in the prediction of a cosmic catastrophe in which the signs of the zodiac are mixed up and the universe is turned upside down. It was not difficult for contemporaries to discern in this chaotic apocalyptic text an allegory for its time. After the only heir to Henry I, William Adeline, drowned in 1120, a dynastic crisis began in England, which, after the death of Henry himself in 1135, escalated into a civil war. nine0003
The period of chaos and lawlessness entered the history of the island under the name «Anarchy». The early readers of Galfrid must have seen Merlin’s prophecies as a warning.
In this enigmatic, chaotic text, two features of Merlin are striking.
Firstly, he is a boy, and secondly, a prophet and clairvoyant. Similar motifs are characteristic of Welsh legends. Greedy for vivid images, Geoffrey picked them up like a magpie picks up shiny objects and adapted them to the story, even though they were not originally connected with Merlin. Galfrid entirely borrowed the image of the prodigy prophet from a Welsh source and renamed him Merlin (lat. Merlinus). In the original, the boy’s name was Ambrose — in Welsh, Emrys. We have already met him in the work already known to the reader — «History of the Britons», written by an anonymous priest who lived in Gwynedd around the 830s and spoke Welsh. nine0003
The story of Ambrose (Ambrosius, Emrys) is one of the most famous Welsh legends and deserves special attention. Before us, no doubt, is a fragment of a folklore tradition included in the text of the History of the Britons. At the center of the story is the tyrant Vortigern (Gvortigirn), who ruled Britain around the same time that the Saxon invasion began.
He allowed them to settle on his lands in exchange for the hand of the daughter of the Saxon leader Hengist. And he made a mistake, because soon the Saxons began to threaten him. Things went from bad to worse when Vortigern married his own daughter and the result of the incestuous relationship was the birth of a son, which drew sharp condemnation from St. Herman, the most influential Christian preacher in Britain at the time. Vortigern attempted to build a shelter in the mountains of Snowdonia, but everything built during the day was destroyed at night by mysterious forces. The author of the History of the Britons continues:
And Gvortigirn, again calling the sorcerers to him, asked them what was the cause of this misfortune and what was happening here. They replied: “Until you find a child who has no father, and this child is not killed, and the ground on which you are going to build a fortress is sprinkled with his blood, the fortress will in no way ever be built.
”
And heeding their advice, he sent out messengers throughout Britain to look for the child who had no father. Having bypassed many regions and regions, they reached the Elleti Plain, which is located in the region called Glegwissing. Here they noticed teenagers playing ball. A quarrel suddenly broke out between the two of them, and one shouted to the other: “Oh, man who has no father, you cannot get the better of me!” Then the messengers carefully questioned the boy and all the others and began to ask his mother if her son had a father. To this she answered in the negative: «I myself do not know how he was conceived in my womb, I know only one thing, that in all my life I have not known a man.» And she swore to them that the boy had no father. And the messengers of Gvortigirn took the boy with them and brought him before the presence of the king. nine0003
Vortigern’s sorcerers are undoubtedly his advisers, who play the same part in the story as the druids in the lives of the early Irish saints; perhaps the motif of the tyrant and the wise men was borrowed from Irish sources.
However, the fatherless boy has a much more powerful prophetic gift than they do. Like the twelve-year-old Jesus, who, according to the Gospel of Luke, surpassed the minds of learned men in the Jerusalem temple, the boy who called himself Ambrose puts the sorcerers to shame. He orders Vortigern’s men to dig where the fortress should have been. A reservoir is found underground, and in it are two vats (apparently communicating), and in them is a canvas in which they found two snakes (or dragons). The boy asks the sorcerer-advisers what this miraculous omen means, but they do not know the answer. Then the boy turns to the king:
This secret is revealed to me, and I will introduce you to it: the canvas is the image of your kingdom, O king; two snakes, then two dragons; the scarlet dragon is yours, and the pond is the image of our world. The white dragon is the dragon of a people who have conquered many regions in Britain and subjugated many peoples living in it, and he will rule in it almost from sea to sea.
But in the future, our people will rise up and throw the people of the Angles over the sea.
The monumental prophecy of the struggle of the native Britons against the alien Saxons has been echoing for centuries; the same theme was raised in Welsh prophetic poetry. When Henry Tudor, a Welsh by birth, assumed the English throne in 1485, Welsh poets proclaimed that he was destined to fulfill an ancient prophecy. The red dragon of the British is still depicted on the flag of Wales. nine0003
Geoffrey of Monmouth borrowed the story of the boy prophet Ambrose from the History of the Britons, renaming him Ambrose of Merlin. Later he included the work «Merlin’s Prophecy» in the main work of his life, «History of the Kings of Britain». So Merlin was first connected with the legends of Arthur.
Two conclusions can be drawn from all this. First, Geoffrey of Monmouth was familiar with the History of the Britons. Secondly, he knew that a character named Merlin (Wall.
Mirdin, Myrddin) was considered a great seer in the Welsh tradition, however, he may not have had more detailed information about him. Galfrid even renamed it. The Latin version of the Welsh Myrddin would look like Merdinus — not the most euphonious name for the French-speaking Anglo-Norman nobility. It sounded too much like the French swear word merde («shit»), so Galfrid subtly replaced «d» with «l». nine0003
However, towards the middle of the 12th century, Geoffrey again turned to the image of Merlin. In the twenty years that have passed since the creation of Merlin’s Prophecies, he collected much information about Myrdin from ancient Welsh sources and embodied them in the poem «The Life of Merlin» (Vita Merlini), written in impeccable and refined Latin. Thanks to this work, we have access to a much older legend about the Celtic savage. His story «wandered» between Ireland, South Scotland and Wales in at least three different ways. All of them acquired literary form at about the same time — in the 12th century.
nine0003
Geoffrey of Monmouth’s Life of Merlin consists of one thousand five hundred and twenty nine lines. Let’s try to briefly outline its confusing content. In the poem, Merlin is the prophet and ruler of the kingdom of Dyfed in southwest Wales. Together with Peredur, King of Gwynedd, and Rydderch, King of Strathclyde, he fights in an unnamed battle against Gwenddoleu, King of Scots. In the book, Geoffrey gives all places and people Latinized names. I use their Welsh forms so that later it will be easier to compare the poem with other sources. Gwenddoleu is defeated, but Merlin is so shocked by the massacre that has taken place that he loses his mind. He flees to the Caledonian forest and lives there, eating grass and fruits. His sister Gwendid, wife of King Ridderch, learns of her brother’s whereabouts and sends a messenger to look for him. The messenger meets Merlin, who complains about the severe winter cold, and in response sings to him about the sadness of Gwendid and Gwendolen, his wife (unbelievable, but Merlin has a wife!).
The song calms Merlin, in a state of enlightenment he agrees to visit his sister. nine0003
But once at the court of King Ridderh, he again falls into madness, and he is chained up to prevent him from returning to the forest.
What follows is a series of events in which Merlin’s madness serves as a source of bitter and ironic insights. Merlin sees the leaf stuck in his sister’s hair and laughs, but refuses to explain the reason for the laugh until he is released. The request is fulfilled, and then he tells Ridderch that the leaf got into Gwendid’s hair when she was reclining with her lover in the open. A shamed Gwendid denies Merlin’s clairvoyant gift. She sends three boys to him in turn — but in fact the same one in different clothes — and demands that his brother predict how each of them will die. The first time, Merlin replies that the boy will die by falling off a cliff, the second time, that he will die in a tree, and the third time, in a river. Ridderh understands that his predictions should not be trusted.
Merlin is then asked if his wife can remarry. He answers in the affirmative, but warns that the future husband should beware of him. Further, Galfrid tells about the fate of the boy: he fell off a cliff, got entangled in the branches of a tree that grew down the slope, and hung so that his head was in the water. All this proves that Merlin is right. nine0003
Returning to the forest, Merlin reads from the stars that Gwendolen is getting married again. He breaks into the wedding riding a deer. Having torn out the horns of the poor animal, he throws them at the groom and kills him, after which he again becomes a prisoner at the court of Ridderkh. There he sees first a beggar, and then a young man buying leather to mend his shoes, and laughs again. Ridderh promises him freedom if he explains the reason for the laughter, and Merlin replies that the beggar, unknowingly, stood on the buried treasure, and the young man will drown before he has time to put on his mended shoes. His predictions come true, and Ridderh lets him go.
nine0003
In the forest, Merlin makes astrological observations in the tower built for him by his sister and predicts the future of Britain until the Norman Conquest.
Ridderch dies and Gwendid mourns for him. Then the «legendary» Taliesin, familiar to us, appears and goes to the forest to visit Merlin. They have learned conversations on a variety of topics, including cosmography, islands, and, oddly enough, fish.
Suddenly, out of nowhere, a spring appears, drinking from which, Merlin gets rid of madness. He thanks God for the healing, and the know-it-all Taliesin takes the opportunity to talk about springs and springs around the world. Upon learning that Merlin is healthy, several rulers visit him in the forest and persuade him to return to the throne, but he refuses. A madman appears, whom Merlin recognizes as a friend of Maeldin’s youth. He lost his mind after eating poisoned apples intended for Merlin. He drinks from the fountain and is also healed.
The four of them Maeldin, Taliesin, Merlin and Gwendid remain in the forest and never again go out into the outside world. The poem ends with the prophecy of Gwendid. She describes in detail the events that are destined to take place in the reign of King Stephen, after which Merlin solemnly renounces his gift of foresight. nine0003
See also
Dog meat, cosmic bulls and ancient historians: what can be said about the Celtic religion in medieval Irish literature or why (neo)druids should not be trusted
The Life of Merlin is an exquisite and wise poem, touching on many topics that interested the intellectuals of the middle of the XII century. Among them, for example, astrology is a relatively new art for those times, borrowed by the Western world from the Arabs. But since many details of the work refer to Welsh and Irish literature, it can be concluded that Geoffrey was familiar with the legend of the Britons about the savage and the prophet, which he turned into a witty poem in Latin.
This theory is confirmed by other sources from medieval Scotland and Wales. The difficulty is that they call the main character in different ways. To avoid confusion, I will use the definition «savage». If we combine these sources and find in them intersections with the «Life of Merlin» by Geoffrey of Monmouth, then we can imagine an approximate plot of the ancient legend of the Britons. The action takes place on the territory of modern South Scotland. The savage was once a king or noble, somehow connected with King Rydderch of Strathclyde. He took part in the battle known as the Battle of Arvderydd between King Ridderch and King Gwenddoleu of Scotland. The Annals of Wales date this battle to 573; it probably took place near Arturet near Carlisle in Cumbria. During the battle, the savage lost his mind and fled to the Caledonian forest, causing the wrath of Ridderh. There he spent many years in horror and madness, along with other hermits. However, his mental breakdown awakened in him a prophetic gift.
The savage’s sister tried to calm him down and bring him back to the castle. Sometimes he let out a strange laugh, followed by predictions, invariably accurate. In the end, he died a «triple death»: pierced, drowned and fallen from a height or hanged. nine0003
Why, when and how to use multithreading and multiprocessing in Python / Habr
Salute, Khabrovites. Right now, OTUS is open for admission to the course «Machine Learning», in connection with this we have translated for you one very interesting «fairy tale». Go.
A long time ago, in a galaxy far, far away…
There lived a wise and powerful wizard in a small village in the middle of the desert. And his name was Dumbledolf. He was not only wise and powerful, but also helped people who came from distant lands to ask for help from a wizard. Our story began when a traveler brought a magic scroll to a wizard.
The traveler did not know what was in the scroll, he only knew that if someone could reveal all the secrets of the scroll, then it was Dumbledolf. nine0003
Chapter 1: Single Thread, Single Process
If you haven’t guessed yet, I made an analogy with the processor and its functions. Our wizard is the processor, and the scroll is a list of links that lead to the power of Python and the knowledge to master it.
The first thought of the wizard, who had no difficulty in deciphering the list, was to send his faithful friend (Harrigorn? I know, I know that sounds terrible) to each of the places that were described in the scroll to find and bring what he can there find. nine0003
In[1]: import urllib.request from concurrent.futures import ThreadPoolExecutor In[2]: urls = [ 'http://www.python.org', 'https://docs.python.org/3/', 'https://docs.python.org/3/whatsnew/3.7.html', 'https://docs.python.org/3/tutorial/index.html', 'https://docs.python.org/3/library/index.html', 'https://docs.python.org/3/reference/index.html', 'https://docs.python.org/3/using/index.html', 'https://docs.python.org/3/howto/index.html', 'https://docs.python.org/3/installing/index.html', 'https://docs.python.org/3/distributing/index.html', 'https://docs.python.org/3/extending/index.html', 'https://docs.python.org/3/c-api/index.html', 'https://docs.python.org/3/faq/index.html' ] In[3]: %%time results = [] for url in urls: with urllib.request.urlopen(url) as src: results.append(src) CPU times: user 135 ms, sys: 283 µs, total: 135 ms Wall time: 12.3s In[]: nine0089
As you can see, we just loop through the URLs one by one with a for loop and read the response. Thanks to %%time and magic IPython , we can see that with my sad internet it took about 12 seconds.
Chapter 2: Multithreading
It was not for nothing that the wizard was famous for his wisdom, he was quickly able to come up with a much more effective way.
Instead of sending one person to each place in order, why not gather a squad of reliable associates and send them to different parts of the world at the same time! The wizard will be able to combine all the knowledge that they bring at once! nine0003
That's right, instead of looping through the list sequentially, we can use multithreading to access multiple URLs at the same time.
In[1]: import urllib.request from concurrent.futures import ThreadPoolExecutor In[2]: urls = [ 'http://www.python.org', 'https://docs.python.org/3/', 'https://docs.python.org/3/whatsnew/3.7.html', 'https://docs.python.org/3/tutorial/index.html', 'https://docs.python.org/3/library/index.html', 'https://docs.python.org/3/reference/index.html', 'https://docs.python.org/3/using/index.html', 'https://docs.python.org/3/howto/index.html', 'https://docs.python.org/3/installing/index.html', 'https://docs.python.org/3/distributing/index.html', 'https://docs.python.org/3/extending/index.html', 'https://docs.python.org/3/c-api/index.html', 'https://docs.python.org/3/faq/index.html' ] In[4]: %%time with ThreadPoolExecutor(4) as executor: results = executor.map(urllib.request.urlopen, urls) CPU times: user 122 ms, sys: 8.27 ms, total: 130 ms Wall time: 3.83s In[5]: %%time with ThreadPoolExecutor(8) as executor: results = executor.map(urllib.request.urlopen, urls) CPU times: user 122 ms, sys: 14.7 ms, total: 137 ms Wall time: 1.79s In[6]: %%time with ThreadPoolExecutor(16) as executor: results = executor.map(urllib.request.urlopen, urls) CPU times: user 143 ms, sys: 3.88 ms, total: 147 ms Wall time: 1.32s In[]:
Much better! Almost like... magic. Using multiple threads can greatly speed up many I/O tasks. In my case, most of the time spent reading URLs is due to network delays. I/O-bound programs spend most of their lifetime waiting for, you guessed it, input or output (much like a wizard waiting for his friends to travel to and from a scroll location).
This can be I/O from the network, a database, a file, or from a user. This I/O typically takes a long time because the source may need to do pre-processing before passing the data to I/O. For example, the processor will count much faster than the network connection will transfer data (about the speed of Flash against your grandmother). nine0003
Note :
multithreadingcan be very useful for tasks like scraping web pages.Chapter 3: Multiprocessing
As the years passed, the fame of the good wizard grew, and with it grew the envy of one hard-hitting dark wizard (Sarumort? Or maybe Voldeman?). Armed with immeasurable cunning and driven by envy, the dark wizard cast a terrible curse on Dumbledolf. When the curse caught up with him, Dumbledolf realized that he only had a few moments to repel it. In desperation, he rummaged through his spell books and quickly found one counter-spell that should have worked.
The only problem was that the wizard had to calculate the sum of all prime numbers less than 1,000,000. A strange spell, of course, but what do we have. nine0003
The wizard knew that calculating the value would be trivial if he had enough time, but he didn't have that luxury. Although he is a great wizard, he is still limited by his humanity and can only check for primeness one number at a time. If he decided to simply sum the prime numbers one after the other, it would take too much time. When only seconds remained before he could cast the counterspell, he suddenly remembered the spell multiprocessing , which I learned from a magic scroll many years ago. This spell will allow him to copy himself in order to distribute the numbers between his copies and check several at the same time. And in the end, all he needs to do is just add up the numbers that he and his copies find.
In[1]: from multiprocessing import Pool In[2]: def if_prime(x): if x <= 1: return 0 elif x <= 3: return x elif x % 2 == 0 or x % 3 == 0: return 0 i = 5 while i**2 <= x: if x % i == 0 or x % (i + 2) == 0: return 0 i += 6 return x In[17]: %%time answer = 0 for i in range(1000000): answer += if_prime(i) CPU times: user 3.48 s, sys: 0 ns, total: 3.48 s Wall time: 3.48s In[18]: %%time if __name__ == '__main__': with Pool(2) as p: answer = sum(p.map(if_prime, list(range(1000000)))) CPU times: user 114 ms, sys: 4.07 ms, total: 118 ms Wall time: 1.91s In[19]: %%time if __name__ == '__main__': with Pool(4) as p: answer = sum(p.map(if_prime, list(range(1000000)))) CPU times: user 99.5 ms, sys: 30.5 ms, total: 130 ms Wall time: 1.12s In[20]: %%timeit if __name__ == '__main__': with Pool(8) as p: answer = sum(p.map(if_prime, list(range(1000000)))) 729 ms ± 3.02 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In[21]: %%timeit if __name__ == '__main__': with Pool(16) as p: answer = sum(p.map(if_prime, list(range(1000000)))) 512ms±39.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In[22]: %%timeit if __name__ == '__main__': with Pool(32) as p: answer = sum(p.map(if_prime, list(range(1000000)))) 518 ms ± 13.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In[23]: %%timeit if __name__ == '__main__': with Pool(64) as p: answer = sum(p.
map(if_prime, list(range(1000000)))) 621 ms ± 10.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) In [ ]:
Modern processors have more than one core, so we can speed up the execution of tasks using the multiprocessing module multiprocessing . CPU-bound tasks are programs that spend most of their time doing computations on the CPU (trivial math, image processing, etc.). If calculations can be performed independently of each other, we are able to divide them among the available processor cores, thereby obtaining a significant increase in processing speed.
All you have to do is:
- Determine the function to apply
- Prepare a list of elements to which the function will be applied;
- Spawn processes using
multiprocessing.Pool. The number that will be passed toPool()will be equal to the number of spawned processes. Inlining the statementwithensures that all processes are killed after they exit.
- Merge the output from the Pool process using the function
map. The input to map will be a function applied to each element and the list of elements itself. nine0134Note : A function can be defined to perform any task that can be executed in parallel. For example, a function might contain code to write the result of a calculation to a file.
So why do we need to separate
multiprocessingandmultithreading? If you have ever tried to increase the performance of a task on a processor using multithreading, in the end the effect was exactly the opposite. That's just terrible! Let's see how it happened. nine0003Just as a wizard is limited by his human nature and can only calculate one number at a time, Python comes with a thing called Global Interpreter Lock (GIL) . Python will happily let you spawn as many threads as you want, but GIL ensures that only one of those threads is running at any given time.
For an I/O-related task, this situation is perfectly normal. One thread sends a request to one URL and waits for a response, only then this thread can be replaced by another that will send another request to a different URL. Since a thread doesn't have to do anything until it receives a response, it doesn't matter if there is only one thread running at a time. nine0003
For CPU-based tasks, multiple threads are almost as useless as armor nipples. Since only one thread can run per unit of time, even if you spawn several, each of which will be allocated a number to test for simplicity, the processor will still work with only one thread. Essentially, these numbers will still be checked one by one. And the overhead of working with multiple threads will contribute to performance degradation, which you just see when using
multithreadingin tasks running on the processor.To get around this "limitation", we use the module
multiprocessing. Instead of using threads, multiprocessing uses, how shall I put it... multiple processes. Each process gets its own interpreter and memory space, so the GIL won't limit you. In fact, each process will use its own processor core and work with its own unique number, and this will be done simultaneously with the work of other processes. How nice of them! nine0003
You may notice that the CPU load will be higher when you use
multiprocessingcompared to a regular for loop or evenmultithreading. This happens because your program uses not one core, but several. And this is good!Be aware that
multiprocessinghas its own overhead for managing multiple processes, which is usually greater than the overhead of multithreading. (Multiprocessing spawns separate interpreters and assigns each process its own area of memory, so yes!) That is, it's generally better to use the light version of multithreading when you want to get away with it this way (think I/O-bound tasks). But when the calculation on the processor becomes a bottleneck, the time comes for modulemultiprocessing.But remember that with great power comes great responsibility.
If you spawn more processes than your processor can process per unit of time, you will notice that performance will start to drop. This happens because the operating system has to do more work by shuffling processes between processor cores, because there are more processes. In reality, everything can be even more complicated than I told today, but I conveyed the main idea. For example, on my system, performance will drop when the number of processes is 16. This is because my processor has only 16 logical cores. nine0003
Chapter 4: Conclusion
- In I/O-bound tasks,
multithreadingcan improve performance. - For I/O-bound tasks,
multiprocessingcan also improve performance, but the overhead is typically higher than usingmultithreading. - The existence of the Python GIL lets us know that only one thread can be running in a program at any given time.
