Search
1000 results for “r_data_table”
-
“They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is.”*…
Back in 2019, (R)D considered a piece from the remarkable Freeman Dyson on what the biotech revolution could mean (itself further to thoughts in an earlier piece of his). Those thoughts popped back into my mind when I read Quentin Hardy‘s recent recounting of his lunch with a friend…
We’re at an outdoor table in Mission Bay, the wet tech hotspot of San Francisco, home to Biopharma, Biotech, and Techbio research labs, known and emerging, plus big hospitals and research outfits.
Across from my salad and his sandwich, Ashlee sweeps his arm in a big arc across Long Bridge Street, towards all the residential and mixed-use buildings.
“There’s dozens of tiny labs up there,” he says, “somebody’s got a mouse, they’re doing something – growing organs, playing with neurons, injecting them with a virus to change their genetics. All kinds of weird shit. It’s wild, man.”
“All kinds of weird shit” and Ashlee have been intimates for years. They have been good to each other. We met around the time computers started moving from the closet to the cloud, and we both wrote about dirt-cheap satellites, and how cell phone guts were ending up in strange places, changing our world with cheap drones and voluminous data. Back when everything really started changing.
I went to Google to write about how those really big data sets and massive amounts of cloud computation were enabling Artificial Intelligence. Ashlee wrote the first biography of Elon Musk, which took him into Musk’s interests in non-governmental rocketry and neural implants. For many years Bloomberg paid him to do a show called Hello World, where he covered Doomsday preppers, fake meat, Nigerian hackers, and all kinds of strange things. All the creative journalists were jealous of him, not least because they couldn’t touch his talent for finding and admiring this abundance of exotic invention.
He now has his own show, Core Memory, which has unsurpassed reporting on all sorts of cutting-edge robotics, life reprogrammers, amateur space stations, body hackers, and new materials manufacturers. Highly recommended.
Back to his current interests. “I know this guy who’s harvesting rat neurons,” he says, “he talks about using them to power data centers.”…
… We start talking about biohacking and self-medication, all the people shooting up peptides, and the places around town where the kids are mixing their AI with their biohacking, and all the quasi-legal stuff people are doing, growing new human and animal parts.
On one level, they’re just following the “lots of data, lots of compute” model, only into the infinitely more complex wet world. Just as enough people posted tagged photos online to enable Fei Fei Li to make and exploit ImageNet, a major milestone in the creation of image-recognition AI, so these new hackers hope to tag, track, remix and scan enough biological data to remake biological understanding. And capability.
I’ve got my kale and he’s got his meat, partly liberated from the bread. Some of the fun in hanging out with Ashlee is the way we can free-associate over years of covering this kind of stuff, knowing that some things blow up and some things don’t work out, good ideas go down while the mad and the lucky are proclaimed geniuses. In other words, we get to bullshit about the weird shit.
“Maybe it’s going to turn into some kind of ghost gun thing, where people take drugs and perform genetic procedures that are legal on their own and turn it into some kind of illegal treatment,” I say. “You’ll go on a luxury cruise into international waters to get your genetic makeup altered, or blend two different animals into a third. Like ‘The Floating Offshore Platform of Dr. Moreau,’” after the H.G. Wells’ story about a mad scientist making human-animal hybrids…
… But we’re also talking about Biology, that most intimate and complex of sciences, being colonized by a trend we’ve seen elsewhere in tech for years: Prices fall far enough to change the rules of access, newcomers hack the system in defiance of the old standards and business models. Oceans of new data turn up, changing the entire process of understanding.
We’ve seen it happen in enough places to know the pattern. Open source Linux, cheap and attractive enough for all kinds of people to improve it for free, wiped out the old computer server industry. WiFi was open source too, so the price was right and interest surged.
The tech doesn’t have to be open source, or free, either. Economic cycles play a part. When the Internet bubble burst, space companies like Iridium and Globalstar, Rotary Rocket and Kistler, crashed. Lots of cheap talent and parts hit the market, which enabled Elon to do Space X. I once did a story about how entertainment in Africa changed after the price of satellite dishes fell below $200, and the tech moved from expatriate compounds to local bars.
The cost of biological experimentation is on a far crazier decline, giving Ashlee a lot of material. Twenty-three years after the first human genome was sequenced at a cost of $2.7 billion, a “complete genetic engineering home lab,” with a refurbished DNA sequencing machine and a “Bioengineering 101 Course” can be yours for $2500. Neurotechnology tools are available for sale or rent, so you can try neural implants at home. China is spinning up dozens of brain-computer interface startups.
“They’ve got a city in China that’s just doing brain technology stuff,” says Ashlee. When I lived in Asia 30 years ago, cities in China were famous for specializing in things like athletic socks and bras, wiping out the competition worldwide by cranking out more stuff more cheaply than anyone else. Now that the abundance of data and the cheapness of commute have kicked off the AI revolution, they have turned to brain tech. I pick at my kale.
Of course, just because the prices are a fraction of what they used to be, and these new hackers are descending on San Francisco, Cambridge, Miami, and who knows where else, it doesn’t mean breakthroughs are at hand. Biology is a lot more complex than electronics – a lot. Perhaps even more important, the new AI technology that people hope will enable all kinds of bio breakthroughs requires enormous amounts of data. The data set has to be huge, it has to be gathered in a single place the AI can access, and perhaps most critically of all, it has to be standardized to the highest quality…
… The biohackers face a big quality issue too. The Nobel Prize-winning protein information made use of some of the cleanest data possible, and Waymo came out of Alphabet’s cutting-edge sensor- and data-analysis labs. The guy in some converted Apartment 3G doing the thing with the iguana liver, the woman in the co-working space with the rat pituitary, they’re probably not going to bring the same magic.
“Yeah, but they’re not the only ones doing this,” says Ashlee. “I just had on Jennifer Doudna.” Doudna, who won a Nobel prize for her work on gene editing, now runs the Innovative Genomics Institute, a place rigorously pursuing this knowledge following traditional standards. She makes a couple of excellent points in Ashlee’s interview. She thinks a lot of the gunslinger biohackers will find biology much more complex and problematic than they think. At the same time, she expects a lot of the regulatory hurdles to new ways of doing things will become familiar over time, lowering the steps and costs of bringing out new drugs and treatments.
These lower costs will make more things possible, and attract more innovation. This will drive crazy a health and insurance industry built around high costs. If history is any guide, the incumbents won’t surrender their high-cost businesses without a fight. That may be one reason why Doudna thinks that big genetic alterations, will show up in agriculture first…
… Which, apparently, at this point isn’t weird enough. “I’ve got to catch up with this university researcher I met at a party,” he says, pushing away his plate. “She’s working on transplanting the personality of one animal, like a dolphin, to another, like a cow.”
“You mean, like you get a cow that wants to body surf in the wake of a tourist boat?”
He nods. “I know. Weird shit, right?”
I barely know what to do with this one, but I’m still in my “Dr. Moreau” zone.
“So maybe someday, instead of capital punishment, a convicted murder will receive the personality of a Labrador Retriever?”“Could be,” he says. “Who knows what people do with this stuff.”
“Has there ever been a time when people were creating a future this weird, when people were going to live in ways they couldn’t even recognize?”
“I dunno,” he says. “Explorer times?”
“I mean yeah, maybe for the Aztecs at first, when they saw the Conquistadors on their horses and thought it was some new kind of hybrid god/animal. But pretty soon the Spanish guys got off their horses and just started messing up the city and killing people. Pretty much like the Aztecs had been doing for a couple of generations. Business as usual.”
“I feel you,” he says. “Hey, I got to go. There’s some guys in Argentina who have this satellite and space tug that went off course. It’s like 50 million kilometers from Earth, but they think they can bring it back.” Weird stuff…
Biohacking in SF, where Dr. Moreau’s a piker, & humanoid robots are a happy delusion. Eminently worth reading in full: “Kale Salad with Ash.”
For more on the dizzying pace of experimentation (this time, in AI), pair with “Agent Claw.”
[Image above: source]
* “At such times the universe gets a little closer to us. They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is. These times are not necessarily good, and not necessarily bad. In fact, what they are depends on what we are.” – Terry Pratchett, I Shall Wear Midnight
###
As we FAFO, we might recall that it was on this date in 1897 that the Indiana State House of Representatives passed Bill No.246 which gave pi the exact value of 3.2– a nice, round– and wrong– number.
Hoosier Dr. Edwin J. Goodwin, M.D, a mathematics enthusiast, satisfied himself that he’d succeeded in “squaring the circle.” Hoping to share with his home state the fame that would surely be forthcoming, Dr. Goodwin drafted legislation that would make Indiana the first to declare the value of pi as law, and convinced Representative Taylor I. Record, a farmer and lumber merchant, to introduce it. As an incentive, Dr. Goodwin, who planned to copyright his “discovery,” offered in the bill to make it available to Indiana textbooks at no cost.
It seems likely that few members of the House understood the bill (many said so during the debate), crammed as it was with 19th century mathematical jargon. Indeed, as Peter Beckmann wrote in his History of Pi, the bill contained “hair-raising statements which not only contradict elementary geometry, but also appear to contradict each other.” (Full text of the bill here.) Still, it sailed through the House.
As it happened, Professor Clarence Abiathar Waldo, the head of the Purdue University Mathematics Department and author of a book titled Manual of Descriptive Geometry, was in the Statehouse lobbying for the University’s budget appropriation as the final debate and vote were underway. He was astonished to find the General Assembly debating mathematical legislation. Naturally, he listened in… and he was horrified.
On February 11 the legislation was introduced in the Senate and referred to the Committee on Temperance, which reported the bill favorably the next day, and sent it to the Senate floor for debate.
But Professor Waldo had “coached” (as he later put it) a number of key Senators on the bill, so this time its reception was different. According to an Indianapolis News report of February 13,
…the bill was brought up and made fun of. The Senators made bad puns about it, ridiculed it and laughed over it. The fun lasted half an hour. Senator Hubbell said that it was not meet for the Senate, which was costing the State $250 a day, to waste its time in such frivolity. He said that in reading the leading newspapers of Chicago and the East, he found that the Indiana State Legislature had laid itself open to ridicule by the action already taken on the bill. He thought consideration of such a propostion was not dignified or worthy of the Senate. He moved the indefinite postponement of the bill, and the motion carried.
As one watches state governments around the U.S. enacting similarly nonsensical, unscientific legislation (e.g., here… perhaps legislators went to school on this), one might be forgiven for wondering “Where’s Waldo?”
#biohacking #ClarenceWaldo #culture #FreemanDyson #history #Indiana #Mathematics #pi #QuentinHardy #Science #Technology -
“They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is.”*…
Back in 2019, (R)D considered a piece from the remarkable Freeman Dyson on what the biotech revolution could mean (itself further to thoughts in an earlier piece of his). Those thoughts popped back into my mind when I read Quentin Hardy‘s recent recounting of his lunch with a friend…
We’re at an outdoor table in Mission Bay, the wet tech hotspot of San Francisco, home to Biopharma, Biotech, and Techbio research labs, known and emerging, plus big hospitals and research outfits.
Across from my salad and his sandwich, Ashlee sweeps his arm in a big arc across Long Bridge Street, towards all the residential and mixed-use buildings.
“There’s dozens of tiny labs up there,” he says, “somebody’s got a mouse, they’re doing something – growing organs, playing with neurons, injecting them with a virus to change their genetics. All kinds of weird shit. It’s wild, man.”
“All kinds of weird shit” and Ashlee have been intimates for years. They have been good to each other. We met around the time computers started moving from the closet to the cloud, and we both wrote about dirt-cheap satellites, and how cell phone guts were ending up in strange places, changing our world with cheap drones and voluminous data. Back when everything really started changing.
I went to Google to write about how those really big data sets and massive amounts of cloud computation were enabling Artificial Intelligence. Ashlee wrote the first biography of Elon Musk, which took him into Musk’s interests in non-governmental rocketry and neural implants. For many years Bloomberg paid him to do a show called Hello World, where he covered Doomsday preppers, fake meat, Nigerian hackers, and all kinds of strange things. All the creative journalists were jealous of him, not least because they couldn’t touch his talent for finding and admiring this abundance of exotic invention.
He now has his own show, Core Memory, which has unsurpassed reporting on all sorts of cutting-edge robotics, life reprogrammers, amateur space stations, body hackers, and new materials manufacturers. Highly recommended.
Back to his current interests. “I know this guy who’s harvesting rat neurons,” he says, “he talks about using them to power data centers.”…
… We start talking about biohacking and self-medication, all the people shooting up peptides, and the places around town where the kids are mixing their AI with their biohacking, and all the quasi-legal stuff people are doing, growing new human and animal parts.
On one level, they’re just following the “lots of data, lots of compute” model, only into the infinitely more complex wet world. Just as enough people posted tagged photos online to enable Fei Fei Li to make and exploit ImageNet, a major milestone in the creation of image-recognition AI, so these new hackers hope to tag, track, remix and scan enough biological data to remake biological understanding. And capability.
I’ve got my kale and he’s got his meat, partly liberated from the bread. Some of the fun in hanging out with Ashlee is the way we can free-associate over years of covering this kind of stuff, knowing that some things blow up and some things don’t work out, good ideas go down while the mad and the lucky are proclaimed geniuses. In other words, we get to bullshit about the weird shit.
“Maybe it’s going to turn into some kind of ghost gun thing, where people take drugs and perform genetic procedures that are legal on their own and turn it into some kind of illegal treatment,” I say. “You’ll go on a luxury cruise into international waters to get your genetic makeup altered, or blend two different animals into a third. Like ‘The Floating Offshore Platform of Dr. Moreau,’” after the H.G. Wells’ story about a mad scientist making human-animal hybrids…
… But we’re also talking about Biology, that most intimate and complex of sciences, being colonized by a trend we’ve seen elsewhere in tech for years: Prices fall far enough to change the rules of access, newcomers hack the system in defiance of the old standards and business models. Oceans of new data turn up, changing the entire process of understanding.
We’ve seen it happen in enough places to know the pattern. Open source Linux, cheap and attractive enough for all kinds of people to improve it for free, wiped out the old computer server industry. WiFi was open source too, so the price was right and interest surged.
The tech doesn’t have to be open source, or free, either. Economic cycles play a part. When the Internet bubble burst, space companies like Iridium and Globalstar, Rotary Rocket and Kistler, crashed. Lots of cheap talent and parts hit the market, which enabled Elon to do Space X. I once did a story about how entertainment in Africa changed after the price of satellite dishes fell below $200, and the tech moved from expatriate compounds to local bars.
The cost of biological experimentation is on a far crazier decline, giving Ashlee a lot of material. Twenty-three years after the first human genome was sequenced at a cost of $2.7 billion, a “complete genetic engineering home lab,” with a refurbished DNA sequencing machine and a “Bioengineering 101 Course” can be yours for $2500. Neurotechnology tools are available for sale or rent, so you can try neural implants at home. China is spinning up dozens of brain-computer interface startups.
“They’ve got a city in China that’s just doing brain technology stuff,” says Ashlee. When I lived in Asia 30 years ago, cities in China were famous for specializing in things like athletic socks and bras, wiping out the competition worldwide by cranking out more stuff more cheaply than anyone else. Now that the abundance of data and the cheapness of commute have kicked off the AI revolution, they have turned to brain tech. I pick at my kale.
Of course, just because the prices are a fraction of what they used to be, and these new hackers are descending on San Francisco, Cambridge, Miami, and who knows where else, it doesn’t mean breakthroughs are at hand. Biology is a lot more complex than electronics – a lot. Perhaps even more important, the new AI technology that people hope will enable all kinds of bio breakthroughs requires enormous amounts of data. The data set has to be huge, it has to be gathered in a single place the AI can access, and perhaps most critically of all, it has to be standardized to the highest quality…
… The biohackers face a big quality issue too. The Nobel Prize-winning protein information made use of some of the cleanest data possible, and Waymo came out of Alphabet’s cutting-edge sensor- and data-analysis labs. The guy in some converted Apartment 3G doing the thing with the iguana liver, the woman in the co-working space with the rat pituitary, they’re probably not going to bring the same magic.
“Yeah, but they’re not the only ones doing this,” says Ashlee. “I just had on Jennifer Doudna.” Doudna, who won a Nobel prize for her work on gene editing, now runs the Innovative Genomics Institute, a place rigorously pursuing this knowledge following traditional standards. She makes a couple of excellent points in Ashlee’s interview. She thinks a lot of the gunslinger biohackers will find biology much more complex and problematic than they think. At the same time, she expects a lot of the regulatory hurdles to new ways of doing things will become familiar over time, lowering the steps and costs of bringing out new drugs and treatments.
These lower costs will make more things possible, and attract more innovation. This will drive crazy a health and insurance industry built around high costs. If history is any guide, the incumbents won’t surrender their high-cost businesses without a fight. That may be one reason why Doudna thinks that big genetic alterations, will show up in agriculture first…
… Which, apparently, at this point isn’t weird enough. “I’ve got to catch up with this university researcher I met at a party,” he says, pushing away his plate. “She’s working on transplanting the personality of one animal, like a dolphin, to another, like a cow.”
“You mean, like you get a cow that wants to body surf in the wake of a tourist boat?”
He nods. “I know. Weird shit, right?”
I barely know what to do with this one, but I’m still in my “Dr. Moreau” zone.
“So maybe someday, instead of capital punishment, a convicted murder will receive the personality of a Labrador Retriever?”“Could be,” he says. “Who knows what people do with this stuff.”
“Has there ever been a time when people were creating a future this weird, when people were going to live in ways they couldn’t even recognize?”
“I dunno,” he says. “Explorer times?”
“I mean yeah, maybe for the Aztecs at first, when they saw the Conquistadors on their horses and thought it was some new kind of hybrid god/animal. But pretty soon the Spanish guys got off their horses and just started messing up the city and killing people. Pretty much like the Aztecs had been doing for a couple of generations. Business as usual.”
“I feel you,” he says. “Hey, I got to go. There’s some guys in Argentina who have this satellite and space tug that went off course. It’s like 50 million kilometers from Earth, but they think they can bring it back.” Weird stuff…
Biohacking in SF, where Dr. Moreau’s a piker, & humanoid robots are a happy delusion. Eminently worth reading in full: “Kale Salad with Ash.”
For more on the dizzying pace of experimentation (this time, in AI), pair with “Agent Claw.”
[Image above: source]
* “At such times the universe gets a little closer to us. They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is. These times are not necessarily good, and not necessarily bad. In fact, what they are depends on what we are.” – Terry Pratchett, I Shall Wear Midnight
###
As we FAFO, we might recall that it was on this date in 1897 that the Indiana State House of Representatives passed Bill No.246 which gave pi the exact value of 3.2– a nice, round– and wrong– number.
Hoosier Dr. Edwin J. Goodwin, M.D, a mathematics enthusiast, satisfied himself that he’d succeeded in “squaring the circle.” Hoping to share with his home state the fame that would surely be forthcoming, Dr. Goodwin drafted legislation that would make Indiana the first to declare the value of pi as law, and convinced Representative Taylor I. Record, a farmer and lumber merchant, to introduce it. As an incentive, Dr. Goodwin, who planned to copyright his “discovery,” offered in the bill to make it available to Indiana textbooks at no cost.
It seems likely that few members of the House understood the bill (many said so during the debate), crammed as it was with 19th century mathematical jargon. Indeed, as Peter Beckmann wrote in his History of Pi, the bill contained “hair-raising statements which not only contradict elementary geometry, but also appear to contradict each other.” (Full text of the bill here.) Still, it sailed through the House.
As it happened, Professor Clarence Abiathar Waldo, the head of the Purdue University Mathematics Department and author of a book titled Manual of Descriptive Geometry, was in the Statehouse lobbying for the University’s budget appropriation as the final debate and vote were underway. He was astonished to find the General Assembly debating mathematical legislation. Naturally, he listened in… and he was horrified.
On February 11 the legislation was introduced in the Senate and referred to the Committee on Temperance, which reported the bill favorably the next day, and sent it to the Senate floor for debate.
But Professor Waldo had “coached” (as he later put it) a number of key Senators on the bill, so this time its reception was different. According to an Indianapolis News report of February 13,
…the bill was brought up and made fun of. The Senators made bad puns about it, ridiculed it and laughed over it. The fun lasted half an hour. Senator Hubbell said that it was not meet for the Senate, which was costing the State $250 a day, to waste its time in such frivolity. He said that in reading the leading newspapers of Chicago and the East, he found that the Indiana State Legislature had laid itself open to ridicule by the action already taken on the bill. He thought consideration of such a propostion was not dignified or worthy of the Senate. He moved the indefinite postponement of the bill, and the motion carried.
As one watches state governments around the U.S. enacting similarly nonsensical, unscientific legislation (e.g., here… perhaps legislators went to school on this), one might be forgiven for wondering “Where’s Waldo?”
#biohacking #ClarenceWaldo #culture #FreemanDyson #history #Indiana #Mathematics #pi #QuentinHardy #Science #Technology -
“They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is.”*…
Back in 2019, (R)D considered a piece from the remarkable Freeman Dyson on what the biotech revolution could mean (itself further to thoughts in an earlier piece of his). Those thoughts popped back into my mind when I read Quentin Hardy‘s recent recounting of his lunch with a friend…
We’re at an outdoor table in Mission Bay, the wet tech hotspot of San Francisco, home to Biopharma, Biotech, and Techbio research labs, known and emerging, plus big hospitals and research outfits.
Across from my salad and his sandwich, Ashlee sweeps his arm in a big arc across Long Bridge Street, towards all the residential and mixed-use buildings.
“There’s dozens of tiny labs up there,” he says, “somebody’s got a mouse, they’re doing something – growing organs, playing with neurons, injecting them with a virus to change their genetics. All kinds of weird shit. It’s wild, man.”
“All kinds of weird shit” and Ashlee have been intimates for years. They have been good to each other. We met around the time computers started moving from the closet to the cloud, and we both wrote about dirt-cheap satellites, and how cell phone guts were ending up in strange places, changing our world with cheap drones and voluminous data. Back when everything really started changing.
I went to Google to write about how those really big data sets and massive amounts of cloud computation were enabling Artificial Intelligence. Ashlee wrote the first biography of Elon Musk, which took him into Musk’s interests in non-governmental rocketry and neural implants. For many years Bloomberg paid him to do a show called Hello World, where he covered Doomsday preppers, fake meat, Nigerian hackers, and all kinds of strange things. All the creative journalists were jealous of him, not least because they couldn’t touch his talent for finding and admiring this abundance of exotic invention.
He now has his own show, Core Memory, which has unsurpassed reporting on all sorts of cutting-edge robotics, life reprogrammers, amateur space stations, body hackers, and new materials manufacturers. Highly recommended.
Back to his current interests. “I know this guy who’s harvesting rat neurons,” he says, “he talks about using them to power data centers.”…
… We start talking about biohacking and self-medication, all the people shooting up peptides, and the places around town where the kids are mixing their AI with their biohacking, and all the quasi-legal stuff people are doing, growing new human and animal parts.
On one level, they’re just following the “lots of data, lots of compute” model, only into the infinitely more complex wet world. Just as enough people posted tagged photos online to enable Fei Fei Li to make and exploit ImageNet, a major milestone in the creation of image-recognition AI, so these new hackers hope to tag, track, remix and scan enough biological data to remake biological understanding. And capability.
I’ve got my kale and he’s got his meat, partly liberated from the bread. Some of the fun in hanging out with Ashlee is the way we can free-associate over years of covering this kind of stuff, knowing that some things blow up and some things don’t work out, good ideas go down while the mad and the lucky are proclaimed geniuses. In other words, we get to bullshit about the weird shit.
“Maybe it’s going to turn into some kind of ghost gun thing, where people take drugs and perform genetic procedures that are legal on their own and turn it into some kind of illegal treatment,” I say. “You’ll go on a luxury cruise into international waters to get your genetic makeup altered, or blend two different animals into a third. Like ‘The Floating Offshore Platform of Dr. Moreau,’” after the H.G. Wells’ story about a mad scientist making human-animal hybrids…
… But we’re also talking about Biology, that most intimate and complex of sciences, being colonized by a trend we’ve seen elsewhere in tech for years: Prices fall far enough to change the rules of access, newcomers hack the system in defiance of the old standards and business models. Oceans of new data turn up, changing the entire process of understanding.
We’ve seen it happen in enough places to know the pattern. Open source Linux, cheap and attractive enough for all kinds of people to improve it for free, wiped out the old computer server industry. WiFi was open source too, so the price was right and interest surged.
The tech doesn’t have to be open source, or free, either. Economic cycles play a part. When the Internet bubble burst, space companies like Iridium and Globalstar, Rotary Rocket and Kistler, crashed. Lots of cheap talent and parts hit the market, which enabled Elon to do Space X. I once did a story about how entertainment in Africa changed after the price of satellite dishes fell below $200, and the tech moved from expatriate compounds to local bars.
The cost of biological experimentation is on a far crazier decline, giving Ashlee a lot of material. Twenty-three years after the first human genome was sequenced at a cost of $2.7 billion, a “complete genetic engineering home lab,” with a refurbished DNA sequencing machine and a “Bioengineering 101 Course” can be yours for $2500. Neurotechnology tools are available for sale or rent, so you can try neural implants at home. China is spinning up dozens of brain-computer interface startups.
“They’ve got a city in China that’s just doing brain technology stuff,” says Ashlee. When I lived in Asia 30 years ago, cities in China were famous for specializing in things like athletic socks and bras, wiping out the competition worldwide by cranking out more stuff more cheaply than anyone else. Now that the abundance of data and the cheapness of commute have kicked off the AI revolution, they have turned to brain tech. I pick at my kale.
Of course, just because the prices are a fraction of what they used to be, and these new hackers are descending on San Francisco, Cambridge, Miami, and who knows where else, it doesn’t mean breakthroughs are at hand. Biology is a lot more complex than electronics – a lot. Perhaps even more important, the new AI technology that people hope will enable all kinds of bio breakthroughs requires enormous amounts of data. The data set has to be huge, it has to be gathered in a single place the AI can access, and perhaps most critically of all, it has to be standardized to the highest quality…
… The biohackers face a big quality issue too. The Nobel Prize-winning protein information made use of some of the cleanest data possible, and Waymo came out of Alphabet’s cutting-edge sensor- and data-analysis labs. The guy in some converted Apartment 3G doing the thing with the iguana liver, the woman in the co-working space with the rat pituitary, they’re probably not going to bring the same magic.
“Yeah, but they’re not the only ones doing this,” says Ashlee. “I just had on Jennifer Doudna.” Doudna, who won a Nobel prize for her work on gene editing, now runs the Innovative Genomics Institute, a place rigorously pursuing this knowledge following traditional standards. She makes a couple of excellent points in Ashlee’s interview. She thinks a lot of the gunslinger biohackers will find biology much more complex and problematic than they think. At the same time, she expects a lot of the regulatory hurdles to new ways of doing things will become familiar over time, lowering the steps and costs of bringing out new drugs and treatments.
These lower costs will make more things possible, and attract more innovation. This will drive crazy a health and insurance industry built around high costs. If history is any guide, the incumbents won’t surrender their high-cost businesses without a fight. That may be one reason why Doudna thinks that big genetic alterations, will show up in agriculture first…
… Which, apparently, at this point isn’t weird enough. “I’ve got to catch up with this university researcher I met at a party,” he says, pushing away his plate. “She’s working on transplanting the personality of one animal, like a dolphin, to another, like a cow.”
“You mean, like you get a cow that wants to body surf in the wake of a tourist boat?”
He nods. “I know. Weird shit, right?”
I barely know what to do with this one, but I’m still in my “Dr. Moreau” zone.
“So maybe someday, instead of capital punishment, a convicted murder will receive the personality of a Labrador Retriever?”“Could be,” he says. “Who knows what people do with this stuff.”
“Has there ever been a time when people were creating a future this weird, when people were going to live in ways they couldn’t even recognize?”
“I dunno,” he says. “Explorer times?”
“I mean yeah, maybe for the Aztecs at first, when they saw the Conquistadors on their horses and thought it was some new kind of hybrid god/animal. But pretty soon the Spanish guys got off their horses and just started messing up the city and killing people. Pretty much like the Aztecs had been doing for a couple of generations. Business as usual.”
“I feel you,” he says. “Hey, I got to go. There’s some guys in Argentina who have this satellite and space tug that went off course. It’s like 50 million kilometers from Earth, but they think they can bring it back.” Weird stuff…
Biohacking in SF, where Dr. Moreau’s a piker, & humanoid robots are a happy delusion. Eminently worth reading in full: “Kale Salad with Ash.”
For more on the dizzying pace of experimentation (this time, in AI), pair with “Agent Claw.”
[Image above: source]
* “At such times the universe gets a little closer to us. They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is. These times are not necessarily good, and not necessarily bad. In fact, what they are depends on what we are.” – Terry Pratchett, I Shall Wear Midnight
###
As we FAFO, we might recall that it was on this date in 1897 that the Indiana State House of Representatives passed Bill No.246 which gave pi the exact value of 3.2– a nice, round– and wrong– number.
Hoosier Dr. Edwin J. Goodwin, M.D, a mathematics enthusiast, satisfied himself that he’d succeeded in “squaring the circle.” Hoping to share with his home state the fame that would surely be forthcoming, Dr. Goodwin drafted legislation that would make Indiana the first to declare the value of pi as law, and convinced Representative Taylor I. Record, a farmer and lumber merchant, to introduce it. As an incentive, Dr. Goodwin, who planned to copyright his “discovery,” offered in the bill to make it available to Indiana textbooks at no cost.
It seems likely that few members of the House understood the bill (many said so during the debate), crammed as it was with 19th century mathematical jargon. Indeed, as Peter Beckmann wrote in his History of Pi, the bill contained “hair-raising statements which not only contradict elementary geometry, but also appear to contradict each other.” (Full text of the bill here.) Still, it sailed through the House.
As it happened, Professor Clarence Abiathar Waldo, the head of the Purdue University Mathematics Department and author of a book titled Manual of Descriptive Geometry, was in the Statehouse lobbying for the University’s budget appropriation as the final debate and vote were underway. He was astonished to find the General Assembly debating mathematical legislation. Naturally, he listened in… and he was horrified.
On February 11 the legislation was introduced in the Senate and referred to the Committee on Temperance, which reported the bill favorably the next day, and sent it to the Senate floor for debate.
But Professor Waldo had “coached” (as he later put it) a number of key Senators on the bill, so this time its reception was different. According to an Indianapolis News report of February 13,
…the bill was brought up and made fun of. The Senators made bad puns about it, ridiculed it and laughed over it. The fun lasted half an hour. Senator Hubbell said that it was not meet for the Senate, which was costing the State $250 a day, to waste its time in such frivolity. He said that in reading the leading newspapers of Chicago and the East, he found that the Indiana State Legislature had laid itself open to ridicule by the action already taken on the bill. He thought consideration of such a propostion was not dignified or worthy of the Senate. He moved the indefinite postponement of the bill, and the motion carried.
As one watches state governments around the U.S. enacting similarly nonsensical, unscientific legislation (e.g., here… perhaps legislators went to school on this), one might be forgiven for wondering “Where’s Waldo?”
#biohacking #ClarenceWaldo #culture #FreemanDyson #history #Indiana #Mathematics #pi #QuentinHardy #Science #Technology -
“They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is.”*…
Back in 2019, (R)D considered a piece from the remarkable Freeman Dyson on what the biotech revolution could mean (itself further to thoughts in an earlier piece of his). Those thoughts popped back into my mind when I read Quentin Hardy‘s recent recounting of his lunch with a friend…
We’re at an outdoor table in Mission Bay, the wet tech hotspot of San Francisco, home to Biopharma, Biotech, and Techbio research labs, known and emerging, plus big hospitals and research outfits.
Across from my salad and his sandwich, Ashlee sweeps his arm in a big arc across Long Bridge Street, towards all the residential and mixed-use buildings.
“There’s dozens of tiny labs up there,” he says, “somebody’s got a mouse, they’re doing something – growing organs, playing with neurons, injecting them with a virus to change their genetics. All kinds of weird shit. It’s wild, man.”
“All kinds of weird shit” and Ashlee have been intimates for years. They have been good to each other. We met around the time computers started moving from the closet to the cloud, and we both wrote about dirt-cheap satellites, and how cell phone guts were ending up in strange places, changing our world with cheap drones and voluminous data. Back when everything really started changing.
I went to Google to write about how those really big data sets and massive amounts of cloud computation were enabling Artificial Intelligence. Ashlee wrote the first biography of Elon Musk, which took him into Musk’s interests in non-governmental rocketry and neural implants. For many years Bloomberg paid him to do a show called Hello World, where he covered Doomsday preppers, fake meat, Nigerian hackers, and all kinds of strange things. All the creative journalists were jealous of him, not least because they couldn’t touch his talent for finding and admiring this abundance of exotic invention.
He now has his own show, Core Memory, which has unsurpassed reporting on all sorts of cutting-edge robotics, life reprogrammers, amateur space stations, body hackers, and new materials manufacturers. Highly recommended.
Back to his current interests. “I know this guy who’s harvesting rat neurons,” he says, “he talks about using them to power data centers.”…
… We start talking about biohacking and self-medication, all the people shooting up peptides, and the places around town where the kids are mixing their AI with their biohacking, and all the quasi-legal stuff people are doing, growing new human and animal parts.
On one level, they’re just following the “lots of data, lots of compute” model, only into the infinitely more complex wet world. Just as enough people posted tagged photos online to enable Fei Fei Li to make and exploit ImageNet, a major milestone in the creation of image-recognition AI, so these new hackers hope to tag, track, remix and scan enough biological data to remake biological understanding. And capability.
I’ve got my kale and he’s got his meat, partly liberated from the bread. Some of the fun in hanging out with Ashlee is the way we can free-associate over years of covering this kind of stuff, knowing that some things blow up and some things don’t work out, good ideas go down while the mad and the lucky are proclaimed geniuses. In other words, we get to bullshit about the weird shit.
“Maybe it’s going to turn into some kind of ghost gun thing, where people take drugs and perform genetic procedures that are legal on their own and turn it into some kind of illegal treatment,” I say. “You’ll go on a luxury cruise into international waters to get your genetic makeup altered, or blend two different animals into a third. Like ‘The Floating Offshore Platform of Dr. Moreau,’” after the H.G. Wells’ story about a mad scientist making human-animal hybrids…
… But we’re also talking about Biology, that most intimate and complex of sciences, being colonized by a trend we’ve seen elsewhere in tech for years: Prices fall far enough to change the rules of access, newcomers hack the system in defiance of the old standards and business models. Oceans of new data turn up, changing the entire process of understanding.
We’ve seen it happen in enough places to know the pattern. Open source Linux, cheap and attractive enough for all kinds of people to improve it for free, wiped out the old computer server industry. WiFi was open source too, so the price was right and interest surged.
The tech doesn’t have to be open source, or free, either. Economic cycles play a part. When the Internet bubble burst, space companies like Iridium and Globalstar, Rotary Rocket and Kistler, crashed. Lots of cheap talent and parts hit the market, which enabled Elon to do Space X. I once did a story about how entertainment in Africa changed after the price of satellite dishes fell below $200, and the tech moved from expatriate compounds to local bars.
The cost of biological experimentation is on a far crazier decline, giving Ashlee a lot of material. Twenty-three years after the first human genome was sequenced at a cost of $2.7 billion, a “complete genetic engineering home lab,” with a refurbished DNA sequencing machine and a “Bioengineering 101 Course” can be yours for $2500. Neurotechnology tools are available for sale or rent, so you can try neural implants at home. China is spinning up dozens of brain-computer interface startups.
“They’ve got a city in China that’s just doing brain technology stuff,” says Ashlee. When I lived in Asia 30 years ago, cities in China were famous for specializing in things like athletic socks and bras, wiping out the competition worldwide by cranking out more stuff more cheaply than anyone else. Now that the abundance of data and the cheapness of commute have kicked off the AI revolution, they have turned to brain tech. I pick at my kale.
Of course, just because the prices are a fraction of what they used to be, and these new hackers are descending on San Francisco, Cambridge, Miami, and who knows where else, it doesn’t mean breakthroughs are at hand. Biology is a lot more complex than electronics – a lot. Perhaps even more important, the new AI technology that people hope will enable all kinds of bio breakthroughs requires enormous amounts of data. The data set has to be huge, it has to be gathered in a single place the AI can access, and perhaps most critically of all, it has to be standardized to the highest quality…
… The biohackers face a big quality issue too. The Nobel Prize-winning protein information made use of some of the cleanest data possible, and Waymo came out of Alphabet’s cutting-edge sensor- and data-analysis labs. The guy in some converted Apartment 3G doing the thing with the iguana liver, the woman in the co-working space with the rat pituitary, they’re probably not going to bring the same magic.
“Yeah, but they’re not the only ones doing this,” says Ashlee. “I just had on Jennifer Doudna.” Doudna, who won a Nobel prize for her work on gene editing, now runs the Innovative Genomics Institute, a place rigorously pursuing this knowledge following traditional standards. She makes a couple of excellent points in Ashlee’s interview. She thinks a lot of the gunslinger biohackers will find biology much more complex and problematic than they think. At the same time, she expects a lot of the regulatory hurdles to new ways of doing things will become familiar over time, lowering the steps and costs of bringing out new drugs and treatments.
These lower costs will make more things possible, and attract more innovation. This will drive crazy a health and insurance industry built around high costs. If history is any guide, the incumbents won’t surrender their high-cost businesses without a fight. That may be one reason why Doudna thinks that big genetic alterations, will show up in agriculture first…
… Which, apparently, at this point isn’t weird enough. “I’ve got to catch up with this university researcher I met at a party,” he says, pushing away his plate. “She’s working on transplanting the personality of one animal, like a dolphin, to another, like a cow.”
“You mean, like you get a cow that wants to body surf in the wake of a tourist boat?”
He nods. “I know. Weird shit, right?”
I barely know what to do with this one, but I’m still in my “Dr. Moreau” zone.
“So maybe someday, instead of capital punishment, a convicted murder will receive the personality of a Labrador Retriever?”“Could be,” he says. “Who knows what people do with this stuff.”
“Has there ever been a time when people were creating a future this weird, when people were going to live in ways they couldn’t even recognize?”
“I dunno,” he says. “Explorer times?”
“I mean yeah, maybe for the Aztecs at first, when they saw the Conquistadors on their horses and thought it was some new kind of hybrid god/animal. But pretty soon the Spanish guys got off their horses and just started messing up the city and killing people. Pretty much like the Aztecs had been doing for a couple of generations. Business as usual.”
“I feel you,” he says. “Hey, I got to go. There’s some guys in Argentina who have this satellite and space tug that went off course. It’s like 50 million kilometers from Earth, but they think they can bring it back.” Weird stuff…
Biohacking in SF, where Dr. Moreau’s a piker, & humanoid robots are a happy delusion. Eminently worth reading in full: “Kale Salad with Ash.”
For more on the dizzying pace of experimentation (this time, in AI), pair with “Agent Claw.”
[Image above: source]
* “At such times the universe gets a little closer to us. They are strange times, times of beginnings and endings. Dangerous and powerful. And we feel it even if we don’t know what it is. These times are not necessarily good, and not necessarily bad. In fact, what they are depends on what we are.” – Terry Pratchett, I Shall Wear Midnight
###
As we FAFO, we might recall that it was on this date in 1897 that the Indiana State House of Representatives passed Bill No.246 which gave pi the exact value of 3.2– a nice, round– and wrong– number.
Hoosier Dr. Edwin J. Goodwin, M.D, a mathematics enthusiast, satisfied himself that he’d succeeded in “squaring the circle.” Hoping to share with his home state the fame that would surely be forthcoming, Dr. Goodwin drafted legislation that would make Indiana the first to declare the value of pi as law, and convinced Representative Taylor I. Record, a farmer and lumber merchant, to introduce it. As an incentive, Dr. Goodwin, who planned to copyright his “discovery,” offered in the bill to make it available to Indiana textbooks at no cost.
It seems likely that few members of the House understood the bill (many said so during the debate), crammed as it was with 19th century mathematical jargon. Indeed, as Peter Beckmann wrote in his History of Pi, the bill contained “hair-raising statements which not only contradict elementary geometry, but also appear to contradict each other.” (Full text of the bill here.) Still, it sailed through the House.
As it happened, Professor Clarence Abiathar Waldo, the head of the Purdue University Mathematics Department and author of a book titled Manual of Descriptive Geometry, was in the Statehouse lobbying for the University’s budget appropriation as the final debate and vote were underway. He was astonished to find the General Assembly debating mathematical legislation. Naturally, he listened in… and he was horrified.
On February 11 the legislation was introduced in the Senate and referred to the Committee on Temperance, which reported the bill favorably the next day, and sent it to the Senate floor for debate.
But Professor Waldo had “coached” (as he later put it) a number of key Senators on the bill, so this time its reception was different. According to an Indianapolis News report of February 13,
…the bill was brought up and made fun of. The Senators made bad puns about it, ridiculed it and laughed over it. The fun lasted half an hour. Senator Hubbell said that it was not meet for the Senate, which was costing the State $250 a day, to waste its time in such frivolity. He said that in reading the leading newspapers of Chicago and the East, he found that the Indiana State Legislature had laid itself open to ridicule by the action already taken on the bill. He thought consideration of such a propostion was not dignified or worthy of the Senate. He moved the indefinite postponement of the bill, and the motion carried.
As one watches state governments around the U.S. enacting similarly nonsensical, unscientific legislation (e.g., here… perhaps legislators went to school on this), one might be forgiven for wondering “Where’s Waldo?”
#biohacking #ClarenceWaldo #culture #FreemanDyson #history #Indiana #Mathematics #pi #QuentinHardy #Science #Technology -
FISA Reauthorization battle heats up – deadline is April 20
It’s that time again! Some FISA Section 702 warrantless wiretapping powers will sunset unless they’re renewed by April 20, so Congress has to do something. The 2024 battle ended up badly for reformers; the two-year reauthorization in the Reforming (Ha-Ha) Intelligence and Securing America Act (RISAA) mostly preserving the status quo and actually expanding surveillance powers. This time, there are multiple reform proposals on the table, although surveillance hawks in both parties are pushing to allow the Trump administration to keep its powers via a “clean” reauthorization.
The GSRA (Government Surveillance Reform Act), sponsored by Senators Ron Wyden (D-Oregon) and Mike Lee (R-Utah) and Representatives Warren Davidson (R-Ohio) and Zoe Lofgren (D-California), is the only bipartisan bicameral reform bill. Sen. Wyden’s website lists the following reforms:
- Closing the backdoor search loophole: The bill requires the federal government to get a warrant to access Americans’ private communications gathered under Section 702, with important exceptions for emergency situations.
- Closing the data broker loophole: The bill bans the federal government from buying Americans’ data from data brokers without a warrant.
- Prohibiting reverse targeting: The bill prohibits using surveillance on foreigners overseas through Section 702 as a pretext for gathering data on Americans.
- Repealing the “make everyone a spy” provision: This bill repeals a controversial 2024 expansion that allows the government to force millions of Americans and companies to secretly spy on its behalf.
- Reforming intelligence collection outside FISA: This bill protects Americans from intelligence agencies using non-statutory authorities, including by prohibiting backdoor searches and reverse targeting outside of FISA.
- Updating privacy protections for AI and other modern technologies: This bill requires federal law enforcement to get a warrant to surveil Americans’ location information, web browsing data, search and chatbot records, and car onboard and telematics data.
- Halting warrantless collection of business records: This bill protects Americans’ data from warrantless collection under an authority that expired over five years ago.
- Enhancing oversight and accountability: The bill strengthens judicial oversight, public reporting, and accountability requirements under FISA.
A wide range of civil rights and digital rights groups endorse the GSRA, including ACLU, Asian Americans Advancing Justice (AAJC), Brennan Center for Justice, CDT, Electronic Privacy Information Center (EPIC), Fight for the Future, Muslim Advocates, estore the Fourth, and Free Press Action. A section-by-section summary of the bill is here. The full bill text is here.
There are also two other reform bills that don’t go quite as far as GSRA, but also have wide support from civil liberties organizations:
- The SAFE act, sponsored by Senator Lee along with Senator Dick Durbin (D-Illinois) is another reform bill, although as EFF notes its reforms fall short in some ways. .
- PLEWSA (the Protect Liberty and End Warrantless Surveillance Act ), from Rep. Andy Biggs (R-Arizona), also includes significant reforms, and may appeal more to Freedom Caucus members in the House.
Meanwhile, House Republican leaders are pushing an 18-month reauthorization without any reforms, a position the White House also supports. But progressive Democrats are opposing reauthorization without a warrant requirement, and there are enough Republican skeptics in the House that it’s not at all clear that a clean reauthorization has the votes – and a recent poll from Demand Progress found that only only 12% percent of voters think Congress should renew FISA without reforms. And to make matters even more complicated, a handful of Republicans are pressing to attach the SAVE Act to the FISA reauth (because hey why not bundle voter suppression with mass surveillance) and a federal judge has ordered the disclosure of FISA noncompliance incident records that the government is trying to keep secret.
So buckle up, it’s likely to be a very interesting month!
“
#FISA #Section702 #surveillance -
Advent Calendar – 2025 – Detail Dialog – Part 1
Classification and objectives from a UI perspective
Today’s Advent Calendar Day focuses specifically on the interaction level prepared in the previous parts. While the basic structure of the user interface and the layout were defined at the beginning, and the interactive table view with sorting, filtering and dynamic actions was subsequently established, it is now a matter of making the transition from overview to detailed observation consistent. The user should no longer only see a tabular collection of data points, but should receive a view tailored to the respective object that enables contextual actions.
- Classification and objectives from a UI perspective
- OverviewView: Interactive enhancements
- Detail view as a standalone UI component (DetailsDialog)
- CreateView: Expiration date in the UI
- Navigation and package structure
- Interaction patterns and UX coherence
The source code for this version can be found on GitHub at https://github.com/svenruppert/url-shortener/tree/feature/advent-2025-day-03
Here’s the screenshot of the version we’re implementing now.
The central goal of this chapter is to improve the user experience without sacrificing the simplicity of what has been done so far. The interface is extended with a detailed view implemented as a standalone dialogue, thereby deliberately creating a clear framework between the overall list and the individual object. This decision follows the principle of the cognitive separation of information spaces: an overview serves to orient and find relevant entries. At the same time, the detailed view creates an isolated, focused working environment.
From an architectural point of view, this extension is an essential step towards a component-oriented UI in which each functional unit – such as creating, displaying or deleting short links – is represented by its own, clearly defined components. The new dialogue includes all the necessary UI elements, validations, and event flows for displaying and interacting with a single ShortUrlMapping object. This decoupling not only enables better testability and reusability but also reduces the cognitive overhead in the primary grid, which previously had to record all interactions directly.
The dialogue serves as an interactive interface between the visual representation and the underlying data model. It reflects the current object, provides copy and navigation actions, and provides visual feedback on the entry’s status – for example, via a colour-coded expiration date. This modular structure allows the user to check details, perform actions, or remove erroneous entries without losing the application’s context.
OverviewView: Interactive enhancements
With the third expansion stage of the user interface, the previous list view is not only expanded, but also functionally upgraded. The OverviewView forms the foundation of the daily work with the created short links and is supplemented in this step by several mechanisms that make the behaviour of the application more natural and efficient.
A central aspect concerns the reactivity of the search fields. In previous versions, search queries were triggered immediately after each input, which was technically correct but inefficient in practice, using the ValueChangeMode.LAZY, in combination with a 400-millisecond delay, the system does not react until the user has completed their input. This delay serves as a natural “pause for thought” and prevents unnecessary updates to the grid. In addition, the enter-key flow has been activated so that a targeted search confirmation can be performed via the keyboard – a typical usage pattern in the professional environment.
codePart.setValueChangeMode(ValueChangeMode.LAZY);codePart.setValueChangeTimeout(400);codePart.addValueChangeListener(e -> refresh());urlPart.setValueChangeMode(ValueChangeMode.LAZY);urlPart.setValueChangeTimeout(400);urlPart.addValueChangeListener(e -> refresh());A second improvement concerns the variety of interactions in the grid. Instead of acting exclusively via buttons, the user can now open a data record by double-clicking or pressing Enter. This form of interaction is not only faster, but also meets the expectations you are used to from desktop applications. To further simplify operation, a context menu has been added that automatically offers the appropriate actions when right-clicking: View details, open target URL, copy shortcode, or delete the entry.
grid.addItemDoubleClickListener(ev -> openDetailsDialog(ev.getItem()));grid.addItemClickListener(ev -> { if (ev.getClickCount() == 2) openDetailsDialog(ev.getItem());});GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid);menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog));menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank")));menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode())));menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));In addition to functionality, the grid column layout has been revised. The shortcode is now displayed in a monospace font, which makes it easier to quickly grasp and compare visually. In addition, a small copy symbol allows copying the entire short link to the clipboard with a mouse click. JavaScript is used to call up the browser’s native clipboard service.
grid.addComponentColumn(m -> { var code = new Span(m.shortCode()); code.getStyle().set("font-family", "ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace"); var copy = new Button(new Icon(VaadinIcon.COPY)); copy.addThemeVariants(ButtonVariant.LUMO_TERTIARY_INLINE); copy.getElement().setProperty("title", "Copy ShortUrl"); copy.addClickListener(_ -> { UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", SHORTCODE_BASE_URL + m.shortCode()); Notification.show("Shortcode copied"); }); var wrap = new HorizontalLayout(code, copy); wrap.setSpacing(true); wrap.setPadding(false); return wrap;}).setHeader("Shortcode").setAutoWidth(true).setFrozen(true).setResizable(true).setFlexGrow(0);In the column with the original URL, the layout has been changed to an elliptical display : Long URLs are truncated, but remain fully visible via the tooltip. This detail improves readability without losing information.
grid.addComponentColumn(m -> { var a = new Anchor(m.originalUrl(), m.originalUrl()); a.setTarget("_blank"); a.getStyle() .set("white-space", "nowrap") .set("overflow", "hidden") .set("text-overflow", "ellipsis") .set("display", "inline-block") .set("max-width", "100%"); a.getElement().setProperty("title", m.originalUrl()); return a;}).setHeader("URL").setFlexGrow(1).setResizable(true);Particularly noteworthy is the new Expires column, which colour-coded status indicators have visually supplemented. These are based on Lumo badges and show the remaining time until a link expires: green for active, yellow for expiring soon, and red for expired.
grid.addComponentColumn(m -> { var pill = new Span(m.expiresAt() .map(ts -> { var days = Duration.between(Instant.now(), ts).toDays(); if (days < 0) return "Expired"; if (days == 0) return "Today"; return "in " + days + " days"; }).orElse("No expiry")); pill.getElement().getThemeList().add("badge pill small"); m.expiresAt().ifPresent(ts -> { long d = Duration.between(Instant.now(), ts).toDays(); if (d < 0) pill.getElement().getThemeList().add("error"); else if (d <= 3) pill.getElement().getThemeList().add("warning"); else pill.getElement().getThemeList().add("success"); }); return pill;}).setHeader("Expires").setAutoWidth(true).setResizable(true).setFlexGrow(0);Finally, the display’s ergonomics have been improved. The grid now works with reduced vertical spacing (compact mode) while retaining its readability thanks to alternating line colours. The line height has been adjusted so that as many entries as possible remain visible even on smaller monitors without making the UI look overloaded.
With these changes, the OverviewView is transformed from a purely management view into a central control tool that provides both a quick overview and deep interaction. This lays the foundation for integrating the detail dialogue – it complements this view with an object-related perspective, while the OverviewView continues to serve as the starting point for navigation.
Detail view as a standalone UI component (DetailsDialog)
Now that the overview list has been expanded with interactive functions, the next logical step is to introduce a standalone UI component focused on the display and management of individual datasets. The goal is to create a modular and reusable view that operates independently of the main view but communicates with it via clearly defined events.
The dialogue is based on Vaadin’s Dialogue class and opens for each selected ShortUrlMapping object. In doing so, it reads out all relevant properties of the passed object – shortcode, target URL, creation time and, optionally, the expiration date. These values are presented in a clearly structured format, supplemented by action buttons to open, copy and delete the entry.
The following excerpt shows the basic structure of the dialogue:
public class DetailsDialog extends Dialog implements HasLogger { public static final ZoneId ZONE = ZoneId.systemDefault(); private static final DateTimeFormatter DATE_TIME_FMT = DateTimeFormatter.ofPattern("dd.MM.yyyy HH:mm").withZone(ZONE); private final String shortCode; private final String originalUrl; private final Instant createdAt; private final Optional<Instant> expiresAt; private final TextField tfShort = new TextField("Shortcode"); private final TextField tfUrl = new TextField("Original URL"); private final TextField tfCreated = new TextField("Created on"); private final TextField tfExpires = new TextField("Expires"); private final Span statusPill = new Span(); private final Button openBtn = new Button("Open", new Icon(VaadinIcon.EXTERNAL_LINK)); private final Button copyShortBtn = new Button("Copy ShortURL", new Icon(VaadinIcon.COPY)); private final Button copyUrlBtn = new Button("Copy URL", new Icon(VaadinIcon.COPY)); private final Button deleteBtn = new Button("Delete...", new Icon(VaadinIcon.TRASH)); private final Button closeBtn = new Button("Close"); public DetailsDialog(ShortUrlMapping mapping) { Objects.requireNonNull(mapping, "mapping"); this.shortCode = mapping.shortCode(); this.originalUrl = mapping.originalUrl(); this.createdAt = mapping.createdAt(); this.expiresAt = mapping.expiresAt(); setHeaderTitle("Details: " + shortCode); setModal(true); setDraggable(true); setResizable(true); setWidth("720px"); openBtn.addThemeVariants(ButtonVariant.LUMO_PRIMARY); deleteBtn.addThemeVariants(ButtonVariant.LUMO_ERROR, ButtonVariant.LUMO_TERTIARY); var headerActions = new HorizontalLayout(openBtn, copyShortBtn, copyUrlBtn, deleteBtn); getHeader().add(headerActions); configureFields(); var form = new FormLayout(); form.add(tfShort, tfUrl, tfCreated, tfExpires, statusPill); form.setColspan(tfUrl, 2); add(form); closeBtn.addClickListener(e -> close()); getFooter().add(closeBtn); wireActions(); }It is already clear here that the dialogue has a high degree of independence. It initialises its contents directly from the passed data object and uses a FormLayout to structure the fields. The input fields are read-only by default because the dialogue is used primarily for display.
The visual feedback on the expiration status is provided by a so-called status pill, which indicates in colour and text whether a short link is still active, about to expire or has already expired. This is done via a small helper method that provides a status description including a colour scheme:
private status computeStatusText() { return expiresAt.map(ts -> { long d = Duration.between(Instant.now(), ts).toDays(); if (d < 0) return new Status("Expired", "error"); if (d == 0) return new Status("Expires today", "warning"); if (d <= 3) return new Status("Expires in " + d + " days", "warning"); return new Status("Valid(" + d + " days left)", "success"); }).orElse(new Status("No expiry", "contrast"));}This status calculation complements the visual feedback from the summary table and provides a consistent representation across the entire application.
The real added value of the DetailsDialog lies in its event-orientation. Instead of the calling view (OverviewView) controlling all actions itself, the actions are defined as Vaadin events. This allows the dialogue to send signals such as OpenEvent, CopyShortcodeEvent, CopyUrlEvent or DeleteEvent to its environment without knowing what they mean:
public static class DeleteEvent extends ComponentEvent<DetailsDialog> { public final String shortCode; public DeleteEvent(DetailsDialog src, String sc) { super(src); this.shortCode = sc; }}In the OverviewView , these events are received and processed:
var dlg = new DetailsDialog(item);dlg.addDeleteListener(ev -> confirmDelete(ev.shortCode));dlg.addOpenListener(ev -> logger().info("Open URL {}", ev.originalUrl));dlg.addCopyShortListener(ev -> logger().info("Copied shortcode {}", ev.shortCode));dlg.addCopyUrlListener(ev -> logger().info("Copied URL {}", ev.url));dlg.open();This creates an apparent decoupling between the presentation and application logic. Dialogue handles representation and interaction; the surrounding view determines how to respond to events.
In summary, the DetailsDialog establishes an architectural principle that goes beyond the specific use case. The combination of a modular UI component, declarative events and a clearly defined data model not only makes the application more flexible, but also more maintainable in the long term. The user benefits from a coherent, clearly structured, detailed view, which makes working with individual entries much more convenient and transparent.
CreateView: Expiration date in the UI
With this step, the creation of new short links receives an important semantic addition: the optional expiration date. The aim is to precisely define the intended service life at the time of creation and to transport this information end-to-end – from the UI to the client to the server and into the persistence.
From a UI point of view, the existing CreateView is extended with DatePicker and TimePicker, flanked by a checkbox labelled “No expiry”. This combination allows both the explicit determination of an end date and the conscious decision to set an unlimited validity. A small but crucial UX rule: the time will remain disabled until a date is selected, and both fields will be disabled if “No expiry” is enabled.
Fields and Basic Configuration
private final TextField urlField = new TextField("Target URL");private final TextField aliasField = new TextField("Alias (optional)");private final Button shortenButton = new Button("Shorten");private final DatePicker expiresDate = new DatePicker("Expires (date)");private final TimePicker expiresTime = new TimePicker("Expires (time)");private final Checkbox noExpiry = new Checkbox("No expiry");private final FormLayout form = new FormLayout();public CreateView() { setSpacing(true); setPadding(true); urlField.setWidthFull(); aliasField.setWidth("300px"); shortenButton.addThemeVariants(ButtonVariant.LUMO_PRIMARY); form.add(urlField, aliasField); configureExpiryFields(); form.setResponsiveSteps( new FormLayout.ResponsiveStep("0", 1), new FormLayout.ResponsiveStep("600px", 2) ); form.setColspan(urlField, 2); var actions = new HorizontalLayout(shortenButton); actions.setAlignItems(Alignment.END); Binder for validations Binder<ShortenRequest> binder = new Binder<>(ShortenRequest.class); ShortenRequest request = new ShortenRequest(); binder.forField(urlField) .asRequired("URL must not be empty") .withValidator(url -> url.startsWith("http://") || url.startsWith("https://"), "Only HTTP(S) URLs allowed") .bind(ShortenRequest::getUrl, ShortenRequest::setUrl); binder.forField(aliasField) .withValidator(a -> a == null || a.isBlank() || a.length() <= AliasPolicy.MAX, "Alias is too long (max " + AliasPolicy.MAX + ")") .withValidator(a -> a == null || a.isBlank() || a.matches(REGEX_ALLOWED), "Only [A-Za-z0-9_-] allowed") .bind(ShortenRequest::getShortURL, ShortenRequest::setShortURL); shortenButton.addClickListener(_ -> { var validated = binder.validate(); if (validated.hasErrors()) return; if (!validateExpiryInFuture()) return; if (binder.writeBeanIfValid(request)) { computeExpiresAt().ifPresent(request::setExpiresAt); var code = createShortCode(request, computeExpiresAt()); code.ifPresentOrElse(c -> { Notification.show("Short link created: " + c); clearForm(binder); }, () -> Notification.show("Alias already assigned or error saving", 3000, Notification.Position.MIDDLE)); } }); add(new H2("Create new short link"), form, actions);}Encapsulating process logic in small auxiliary methods improves readability and testability. The calculation uses the local time zone and provides an instant that can be processed unchanged on the server side.
private static final ZoneId ZONE = ZoneId.systemDefault();private void configureExpiryFields() { expiresDate.setClearButtonVisible(true); expiresDate.setPlaceholder("dd.MM.yyyy"); expiresTime.setStep(Duration.ofMinutes(1)); expiresTime.setPlaceholder("HH:mm"); Activate time only when a date is set expiresTime.setEnabled(false); expiresDate.addValueChangeListener(ev -> { boolean hasDate = ev.getValue() != null; expiresTime.setEnabled(hasDate && !noExpiry.getValue()); }); noExpiry.addValueChangeListener(ev -> { boolean disabled = ev.getValue(); expiresDate.setEnabled(!disabled); expiresTime.setEnabled(!disabled && expiresDate.getValue() != null); }); form.add(noExpiry, expiresDate, expiresTime);}private Optional<Instant> computeExpiresAt() { if (Boolean.TRUE.equals(noExpiry.getValue())) return Optional.empty(); LocalDate d = expiresDate.getValue(); LocalTime t = expiresTime.getValue(); if (d == null || t == null) return Optional.empty(); return Optional.of(ZonedDateTime.of(d, t, ZONE).toInstant());}private boolean validateExpiryInFuture() { var exp = computeExpiresAt(); if (exp.isPresent() && exp.get().isBefore(Instant.now())) { Notification.show("Expiry must be in the future"); return false; } return true;}private void clearForm(Binder<ShortenRequest> binder) { urlField.clear(); aliasField.clear(); noExpiry.clear(); expiresDate.clear(); expiresTime.clear(); binder.setBean(new ShortenRequest()); urlField.setInvalid(false); aliasField.setInvalid(false);}It is important to pass it on transparently to the client. Instead of storing the time in the UI, it is transmitted to the server in the request. To do this, the CreateView calls the client with the extended signature. The client validates only if an alias is set and passes the data to the server unchanged.
private Optional<String> createShortCode(ShortenRequest req, Optional<Instant> expiresAt) { logger().info("createShortCode with ShortenRequest '{}'", req); try { var customMapping = urlShortenerClient.createCustomMapping(req.getShortURL(), req.getUrl(), expiresAt.orElse(null)); return Optional.ofNullable(customMapping.shortCode()); } catch (IllegalArgumentException | IOException e) { logger().error("Error saving", e); return Optional.empty(); }}On the client side, the payload is serialised as a ShortenRequest and sent to the /shorten endpoint. The response is not limited to the shortcode; it is also parsed as a full ShortUrlMapping. As a result, the UI immediately knows the server-side confirmed state – including the expiresAt.
public ShortUrlMapping createCustomMapping(String alias, String url, Instant expiredAt) throws IOException { logger().info("Create custom mapping alias='{}' url='{}' expiredAt='{}'", alias, url, expiredAt); if (alias != null && !alias.isBlank()) { var validate = AliasPolicy.validate(alias); if (!validate.valid()) { var reason = validate.reason(); throw new IllegalArgumentException(reason.defaultMessage); } } URL shortenUrl = serverBaseAdmin.resolve(PATH_ADMIN_SHORTEN).toURL(); HttpURLConnection connection = (HttpURLConnection) shortenUrl.openConnection(); connection.setRequestMethod("POST"); connection.setDoOutput(true); connection.setRequestProperty(CONTENT_TYPE, JSON_CONTENT_TYPE); var shortenRequest = new ShortenRequest(url, alias, expiredAt); String body = shortenRequest.toJson(); try (OutputStream os = connection.getOutputStream()) { os.write(body.getBytes(UTF_8)); } int status = connection.getResponseCode(); if (status == 200 || status == 201) { try (InputStream is = connection.getInputStream()) { String jsonResponse = new String(is.readAllBytes(), UTF_8); ShortUrlMapping shortUrlMapping = fromJson(jsonResponse, ShortUrlMapping.class); return shortUrlMapping; } } if (status == 409) { throw new IllegalArgumentException("Alias already in use"); } throw new IOException("Unexpected status: " + status);}On the server side, the ShortenHandler receives the extended request, validates the required fields, and then assigns the store to create it. The response contains the complete mapping object that the client and UI can use immediately.
final String body = readBody(ex.getRequestBody());ShortenRequest req = fromJson(body, ShortenRequest.class);if (isNullOrBlank(req.getUrl())) { writeJson(ex, BAD_REQUEST, "Missing 'url'"); return;}final Result<ShortUrlMapping> urlMappingResult = store.createMapping(req.getShortURL(), req.getUrl(), req.getExpiresAt());urlMappingResult .ifPresentOrElse(success -> logger().info("mapping created success {}", success.toString()), failed -> logger().info("mapping created failed - {}", failed));urlMappingResult .ifSuccess(mapping -> { final Headers h = ex.getResponseHeaders(); h.add("Location", "/r/" + mapping.shortCode()); writeJson(ex, fromCode(201), toJson(mapping)); }) .ifFailure(errorJson -> { try { var parsed = JsonUtils.parseJson(errorJson); var errorCode = Integer.parseInt(parsed.get("code")); var message = parsed.get("message"); writeJson(ex, fromCode(errorCode), message); } catch (Exception e) { writeJson(ex, CONFLICT, errorJson); } });In summary, a consistent, end-to-end process is created: The user optionally defines an expiration date when making them, the UI validates basic rules, the client transfers the semantics losslessly, and the server persists them reliably. The OverviewView and the DetailsDialog can then display and interpret this information immediately. This adds a central property to the domain without complicating the existing operating flow.
Navigation and package structure
The previous user interface has become much more complex in terms of content due to the detailed dialogue and the extended form functions. To reflect this development, the package structure in the UI module was clearly reorganised. The goal was not only a logical grouping according to responsibilities, but also a long-term basis for extended navigation concepts and modular extensions.
Previously, the OverviewView was still in the general package com.svenruppert.urlshortener.ui.vaadin.views. With the introduction of detailed dialogue and the growing importance of this area in terms of content, it was included in a separate subpackage, “views.overview“, and moved there. This decision not only creates room for additional components (e.g. context menus, helper dialogues or filters), but also follows the principle of functional coherence: All classes that together form the overview function are now centrally bundled.
In the code, this step is reflected in the customisation of the import within the MainLayout :
old:
import com.svenruppert.urlshortener.ui.vaadin.views.OverviewView;
new:
import com.svenruppert.urlshortener.ui.vaadin.views.overview.OverviewView;
This small but significant step marks the transition from a purely page-based structure to a component-oriented structure. The MainLayout continues to serve as the central navigation element of the application; the individual views are now more decoupled and can be further developed independently. This creates a clear separation between layout logic (central navigation, menus, visual frameworks) and functional logic (display, interaction, data flow).
The route relationships are deliberately kept simple. The OverviewView is still registered under the path /overview and uses the MainLayout as its parent layout element:
@PageTitle Overview@Route(value = OverviewView.PATH, layout = MainLayout.class)public class OverviewView extends VerticalLayout implements HasLogger { public static final String PATH = "overview"; // ...}This configuration keeps navigation consistent with the other parts of the application, such as CreateView, AboutView, or YoutubeView. New views can be easily added without affecting the central navigation mechanism. This ensures maintainability and scalability – two central requirements for an application that grows gradually within the Advent calendar framework.
The MainLayout itself is largely unchanged, but has been updated during the package migration to reflect new imports and menu entries. The referencing of the OverviewView is particularly important, as it represents the entry point of user interaction:
SideNavItem overview = new SideNavItem("Overview", OverviewView.class, VaadinIcon.LIST.create());SideNavItem create = new SideNavItem("Create", CreateView.class, VaadinIcon.PLUS.create());SideNavItem about = new SideNavItem("About", AboutView.class, VaadinIcon.INFO_CIRCLE.create());SideNavItem youtube = new SideNavItem("Youtube", YoutubeView.class, VaadinIcon.YOUTUBE.create());SideNav nav = new SideNav(overview, create, about, youtube);addToDrawer(nav);This clean separation between routing, structure and presentation lays the foundation for the further development of the user interface. Future features – such as detailed filters, user preferences or administrative functions – can be easily integrated into their own subpackages and namespaces without affecting core navigation. This is the step towards a sustainable UI architecture that specifically supports both growing complexity and extensibility.
Interaction patterns and UX coherence
As the application’s feature density increases, the importance of a consistent user experience grows. While the first days of the Advent calendar laid the technical foundation, so far, the focus has been on functionality. In this chapter, the focus shifts to the coherence of interaction patterns, i.e. how users interact with the application in a consistent, predictable rhythm.
Central to this is the goal of establishing uniform behaviour for recurring actions. Whether in the overview, in the detail dialogue or in forms – copying, opening or deleting should always feel the same. The application thus conveys reliability, which is particularly crucial for technical users of web-based tools.
Uniform copying behaviour
A good example is copying URLs and short links. The exact mechanisms are used in both the grid and the detail dialogue: a button with the VaadinIcon.COPY symbol: an asynchronous clipboard action via JavaScript and a discreet confirmation message. This avoids users having to learn different forms of interaction in various views.
copy.addClickListener(_ -> { UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", SHORTCODE_BASE_URL + m.shortCode()); Notification.show("Shortcode copied");});Such a detail may seem inconspicuous, but it has a significant impact on the perceived professionalism of the application. The feedback system via notifications plays a central role here: The user receives an immediate, unobtrusive signal about the success of his action – a kind of visual receipt that creates trust.
Context menus and multiple interactions
Another element of UX coherence is the context menu in the summary table. It allows access to the same actions, which can also be accessed via buttons or double-clicks. This redundant but deliberate multiple interaction follows the principle of user freedom: Users can choose whether to act via direct icons, the keyboard or the context menu.
GridContextMenu<ShortUrlMapping> menu = new GridContextMenu<>(grid);menu.addItem("Show details", e -> e.getItem().ifPresent(this::openDetailsDialog));menu.addItem("Open URL", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().open(m.originalUrl(), "_blank")));menu.addItem("Copy shortcode", e -> e.getItem().ifPresent(m -> UI.getCurrent().getPage().executeJs("navigator.clipboard.writeText($0)", m.shortCode())));menu.addItem("Delete...", e -> e.getItem().ifPresent(m -> confirmDelete(m.shortCode())));The decision to use context menus is not only aesthetic but also ergonomic: it reduces the grid’s optical density without sacrificing functionality. Actions only appear when they are needed – an approach that is essential in complex management interfaces.
Consistency of feedback
A uniform pattern is also followed for error messages and validations. The system does not abruptly abort user actions; instead, it clearly communicates why an input was not accepted. Example: The expiration date cannot be in the past. This rule is conveyed both visually and by message.
if (exp.isPresent() && exp.get().isBefore(Instant.now())) { Notification.show("Expiry must be in the future"); return false;}Accurate feedback prevents the user from perceiving the system as unpredictable. Every validation, every hint, and every success message follows the same communicative style—short, clear, and polite.
HTTPS Requirement for Clipboard
A technical but essential detail is that the Clipboard API only works in modern browsers within secure contexts (HTTPS or localhost). Therefore, the copy feature is deliberately designed to fail elegantly when clipboard access is unavailable. The application does not crash, but responds silently – an aspect that underlines the robustness and professionalism of the user experience.
The mechanisms described in this chapter – consistent feedback, redundant operating options and secure fallbacks – together contribute to a coherent user experience. They form the basis of trust and predictability, two characteristics that are essential as software becomes more complex. Overall, the result is an interface that can be operated intuitively, even as its technical depth in the background continues to increase.
Cheers Sven
-
Drop #603 (2025-02-12): Suggest • Static • Search
Zed Edit Predictions (Beta); Data-Driven 11ty Site Generation; Pagefind
The Drop is back after the $WORK-fueled hiatus. Travel to and from San Diego was horrendous, and it took most of the weekend to recover from the ~36 hours it took me to get from San Diego back to the Maine compound. The missed Monday Drop was due to porting 47 Watch to Eleventy/11ty. We’ll talk more about that in one of the sections. Tuesday’s miss was due to the ~57 minutes of slumber between Monday and Tuesday thanks to the still unabated long covid resurgence.
Anyway…
Thanks to said 11ty work, today’s Drop showcases some of the tech used to make the new site.
Type your email…
Subscribe
TL;DR
(This is an AI-generated summary of today’s Drop using Ollama + llama 3.2 and a custom prompt.)
- Zed’s new beta edit predictions feature offers multi-line code completion and cross-language awareness, proving especially useful for mixed-language projects and repetitive code modifications.
- 11ty’s data-driven document generation capability enables efficient site building by pairing templates with JSON data files, eliminating the need for multiple individual markdown files.
- Pagefind provides framework-agnostic static site search functionality with efficient chunked indexing, supporting multiple languages and content types while maintaining small network payloads.
Zed Edit Predictions (Beta)
Zed Completions BetaI’ve almost entirely switched to Zed for everything but R work (where I use RStudio, since I will not suffer Positron on my systems due to the Microsoft corruption); and, I do a great deal of work in Zed every day. Said work usually involves switching between many projects, which may be Go, Rust, Shell, or Java|TypeScript-based, plus a healthy dose of HTML, CSS, and Markdown mixed in.
I make heavy use of most of most of Zed’s features, including the AI Assitant Panel and inline AI assistance (which is incredibly handy in the Zed Alacritty terminal, too). I also let Zed collect telemetry, since I work for a startup who uses quite a bit of telemetry in our Visualizer app (so, “fair’s fair”). I mention that last bit as I guess I use Zed just enough for the Zed team to hit me up to test out their beta edit predictions feature. If you’ve used GitHub Copilot or Cody in VS Code/Codium, you’ve experienced edit predictions there. The TLDR for the feature is that Zed’s “Zeta” model has your file and project context at its disposal, keeps track of what you’ve been typing, and tries to pick what it thinks is the likely desired next bit of code or content. The video in the section header does a fine job showing off the functionality.
I got in the beta right before I started the 11ty conversion of 47 Watch, and I cannot fully descibe how handy it was to perform that port. I haven’t been using VS Codium much, of late, so I don’t know if Cody (et al.) has advanced there enough to have a simialr feature set, but — for starters — Zed’s completion works across multiple source code lines (i.e., it can — with oddly great accuracy — see you’re modifying one tag or bit of code, and can surmize you want to make similar changes to similar lines, and it just “does it” after you tab-complete). In a co-mingled project with a Go microservice back end and HTML/JS/CSS front-end, it has yet to get confused by said source diversity, and consistently suggests proper in-language completions for each file type. In fact, it does an uncanny job suggesting front end route code completions when a new back end REST API handler is added.
This new feature saved me hours during the porting process, and I’m writing this section mostly to suggest (heh) you give Zed a go again (if you’re not using it already), and try to get into the beta.
You can disable it entirely or per-project if you don’t want it showing up in, say, a markdown document like the one I’m editing right now, where I want my own words.
I’ll risk one more bit of extended blather and also note that it is super important to treat all the AI “code for me” tools as actual junior-grade assistants, where you sort of trust the output of the task you’ve given them, but also verify and fully grok said output before using it. And, (finally), I also highly suggest — every so often — doing some of the boring coding you may be offloading to LLM assistants. Think of them as katas that keep your brain grounded in the fundamentals. It’s way too easy to let certain skills atropy if you’re relying on a power tool to do the work for you.
Zed Bonus Section
I snuck in this bonus section since one of the few things that kept me from using Zed for editing the Drop posts was the lack of an HTML-based preview for markdown documents. The built-in previewer is fast and looks great, but I can’t copy from it, paste the contents into the WordPress editor, and have formatted content ready to go.
This Zed task snippet:
{ "label": "markdown preview", "command": "pandoc ${ZED_RELATIVE_FILE} -f markdown -t html --metadata title=' ' --no-highlight -s --include-in-header=/Users/hrbrmstr/.cache/darkmeta.html -o /tmp/markdown-preview.html && open /tmp/markdown-preview.html"},Uses pandoc with a dark-mode template (see below) to provide the functionality I’m used to from VS Codium. We’ll eventually talk more about Zed tasks in a future Drop.
<meta name="color-scheme" content="dark light"><style>* { font-family: sans-serif;}html { color: #e0e0e0; background-color: #1a1a1a;}body { margin: 0 auto; max-width: 36em; padding-left: 50px; padding-right: 50px; padding-top: 50px; padding-bottom: 50px; hyphens: auto; overflow-wrap: break-word; text-rendering: optimizeLegibility; font-kerning: normal;}@media (max-width: 600px) { body { font-size: 0.9em; padding: 12px; } h1 { font-size: 1.8em; }}@media print { html { background-color: white; } body { background-color: transparent; color: black; font-size: 12pt; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3, h4 { page-break-after: avoid; }}p { margin: 1em 0;}a { color: #7eb6ff;}a:visited { color: #b19eff;}img { max-width: 100%;}svg { height: auto; max-width: 100%;}h1, h2, h3, h4, h5, h6 { margin-top: 1.4em; color: #ffffff;}h5, h6 { font-size: 1em; font-style: italic;}h6 { font-weight: normal;}ol, ul { padding-left: 1.7em; margin-top: 1em;}li > ol, li > ul { margin-top: 0;}blockquote { margin: 1em 0 1em 1.7em; padding-left: 1em; border-left: 2px solid #404040; color: #b0b0b0;}code { font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace; font-size: 85%; margin: 0; hyphens: manual; background-color: #2d2d2d; color: #e0e0e0; padding: 0.2em 0.4em; border-radius: 3px;}pre { margin: 1em 0; overflow: auto; background-color: #2d2d2d; padding: 1em; border-radius: 5px;}pre code { padding: 0; overflow: visible; overflow-wrap: normal; background-color: transparent;}.sourceCode { background-color: transparent; overflow: visible;}hr { background-color: #404040; border: none; height: 1px; margin: 1em 0;}table { margin: 1em 0; border-collapse: collapse; width: 100%; overflow-x: auto; display: block; font-variant-numeric: lining-nums tabular-nums;}table caption { margin-bottom: 0.75em;}tbody { margin-top: 0.5em; border-top: 1px solid #404040; border-bottom: 1px solid #404040;}th { border-top: 1px solid #404040; padding: 0.25em 0.5em 0.25em 0.5em;}td { padding: 0.125em 0.5em 0.25em 0.5em;}header { margin-bottom: 4em; text-align: center;}#TOC li { list-style: none;}#TOC ul { padding-left: 1.3em;}#TOC > ul { padding-left: 0;}#TOC a:not(:hover) { text-decoration: none;}code{white-space: pre-wrap;}span.smallcaps{font-variant: small-caps;}div.columns{display: flex; gap: min(4vw, 1.5em);}div.column{flex: auto; overflow-x: auto;}div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}ul.task-list[class]{list-style: none;}ul.task-list li input[type="checkbox"] { font-size: inherit; width: 0.8em; margin: 0 0.8em 0.2em -1.6em; vertical-align: middle;}.display.math{display: block; text-align: center; margin: 0.5rem auto;}</style>Data-Driven 11ty Site Generation
For my 47 Watch project, I edit Executive Orders (et al.) as markdown documents with a particular structure, which makes it easy to turn them into data. After each new edit session in 47 Watch, I rebuild a bunch of data files, including this 11ty master JSON file that contains all the posts nestled up into a nice data frame-esque structure.
While I could have had a script generate dozens (or, eventually hundreds) of 11thy markdown post files I did not have to. All I needed to do was ensure that aforelinked
executive-orders.11tydata.jsonfile was in the right directory and then take advantage of 11ty’s excellent data-driven document generation capability.On
build, 11ty will pair up the base template (in this caseexecutive-order.njk) with that data file, then proceed to generate as many of the files as there are in the array. The index for that collection will also iterate over the array.There are a few Community posts which will help you explore this feature further (or you can clone and play with the 47 Watch code directly).
Pagefind
Pagefind is an open-source library that addresses a common challenge in static site development: providing robust search capabilities without relying on external services or heavy server infrastructure.
I ended up using it for 47 Watch as it has a pretty innovative approach to handling large-scale search operations. The library employs a clever chunking mechanism for its search index, ensuring that folks only download the necessary portions of the index when performing searches. This architectural decision yields some pretty impressive performance metrics — imagine being able to implement full-text search across 10,000 pages while keeping the network payload under 300kB, including the library itself.
The framework-agnostic nature of Pagefind is especially valuable if you’re a gadfly like I am. Whether you’re working with modern frameworks like Next.js and SvelteKit or traditional static site generators like Jekyll and Hugo, Pagefind operates on the final built files, which eliminates the need for framework-specific implementations or affordances.
Pagefind’s also has multilingual support and that plus the rich filtering engine makes it particularly suitable for knowledge bases and research-oriented websites. The ability to customize content weighting and metadata tracking allows for fine-tuned search experiences that can be tailored to specific academic or research needs.
The library’s capability to index non-traditional content types through its NodeJS indexing library — including PDFs and JSON files — extends its utility beyond simple web page searches, making it a versatile tool for comprehensive digital resource management.
If you look at some of the provided demos (MDN, Godot documentation, and XKCD), you can observe for yourself how Pagefind maintains its performance.
All I really had to do was add
"postbuild": "pagefind --site _site",to thepackage.jsonand it worked the first time.I had originally intended to use Algolia, but I’m trying to avoid external third-party dependencies for this project, and Pagefind has all of the features I need for the immediate future.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- 🐘 Mastodon via
@[email protected] - 🦋 Bluesky via
https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy
Also, refer to:
to see how to access a regularly updated database of all the Drops with extracted links, and full-text search capability. ☮️
#1a1a1a #2d2d2d #404040 #7eb6ff #b0b0b0 #b19eff #e0e0e0 #ffffff #TOC
-
Drop #603 (2025-02-12): Suggest • Static • Search
Zed Edit Predictions (Beta); Data-Driven 11ty Site Generation; Pagefind
The Drop is back after the $WORK-fueled hiatus. Travel to and from San Diego was horrendous, and it took most of the weekend to recover from the ~36 hours it took me to get from San Diego back to the Maine compound. The missed Monday Drop was due to porting 47 Watch to Eleventy/11ty. We’ll talk more about that in one of the sections. Tuesday’s miss was due to the ~57 minutes of slumber between Monday and Tuesday thanks to the still unabated long covid resurgence.
Anyway…
Thanks to said 11ty work, today’s Drop showcases some of the tech used to make the new site.
TL;DR
(This is an AI-generated summary of today’s Drop using Ollama + llama 3.2 and a custom prompt.)
- Zed’s new beta edit predictions feature offers multi-line code completion and cross-language awareness, proving especially useful for mixed-language projects and repetitive code modifications.
- 11ty’s data-driven document generation capability enables efficient site building by pairing templates with JSON data files, eliminating the need for multiple individual markdown files.
- Pagefind provides framework-agnostic static site search functionality with efficient chunked indexing, supporting multiple languages and content types while maintaining small network payloads.
Zed Edit Predictions (Beta)
Zed Completions BetaI’ve almost entirely switched to Zed for everything but R work (where I use RStudio, since I will not suffer Positron on my systems due to the Microsoft corruption); and, I do a great deal of work in Zed every day. Said work usually involves switching between many projects, which may be Go, Rust, Shell, or Java|TypeScript-based, plus a healthy dose of HTML, CSS, and Markdown mixed in.
I make heavy use of most of most of Zed’s features, including the AI Assitant Panel and inline AI assistance (which is incredibly handy in the Zed Alacritty terminal, too). I also let Zed collect telemetry, since I work for a startup who uses quite a bit of telemetry in our Visualizer app (so, “fair’s fair”). I mention that last bit as I guess I use Zed just enough for the Zed team to hit me up to test out their beta edit predictions feature. If you’ve used GitHub Copilot or Cody in VS Code/Codium, you’ve experienced edit predictions there. The TLDR for the feature is that Zed’s “Zeta” model has your file and project context at its disposal, keeps track of what you’ve been typing, and tries to pick what it thinks is the likely desired next bit of code or content. The video in the section header does a fine job showing off the functionality.
I got in the beta right before I started the 11ty conversion of 47 Watch, and I cannot fully descibe how handy it was to perform that port. I haven’t been using VS Codium much, of late, so I don’t know if Cody (et al.) has advanced there enough to have a simialr feature set, but — for starters — Zed’s completion works across multiple source code lines (i.e., it can — with oddly great accuracy — see you’re modifying one tag or bit of code, and can surmize you want to make similar changes to similar lines, and it just “does it” after you tab-complete). In a co-mingled project with a Go microservice back end and HTML/JS/CSS front-end, it has yet to get confused by said source diversity, and consistently suggests proper in-language completions for each file type. In fact, it does an uncanny job suggesting front end route code completions when a new back end REST API handler is added.
This new feature saved me hours during the porting process, and I’m writing this section mostly to suggest (heh) you give Zed a go again (if you’re not using it already), and try to get into the beta.
You can disable it entirely or per-project if you don’t want it showing up in, say, a markdown document like the one I’m editing right now, where I want my own words.
I’ll risk one more bit of extended blather and also note that it is super important to treat all the AI “code for me” tools as actual junior-grade assistants, where you sort of trust the output of the task you’ve given them, but also verify and fully grok said output before using it. And, (finally), I also highly suggest — every so often — doing some of the boring coding you may be offloading to LLM assistants. Think of them as katas that keep your brain grounded in the fundamentals. It’s way too easy to let certain skills atropy if you’re relying on a power tool to do the work for you.
Zed Bonus Section
I snuck in this bonus section since one of the few things that kept me from using Zed for editing the Drop posts was the lack of an HTML-based preview for markdown documents. The built-in previewer is fast and looks great, but I can’t copy from it, paste the contents into the WordPress editor, and have formatted content ready to go.
This Zed task snippet:
{ "label": "markdown preview", "command": "pandoc ${ZED_RELATIVE_FILE} -f markdown -t html --metadata title=' ' --no-highlight -s --include-in-header=/Users/hrbrmstr/.cache/darkmeta.html -o /tmp/markdown-preview.html && open /tmp/markdown-preview.html"},Uses pandoc with a dark-mode template (see below) to provide the functionality I’m used to from VS Codium. We’ll eventually talk more about Zed tasks in a future Drop.
<meta name="color-scheme" content="dark light"><style>* { font-family: sans-serif;}html { color: #e0e0e0; background-color: #1a1a1a;}body { margin: 0 auto; max-width: 36em; padding-left: 50px; padding-right: 50px; padding-top: 50px; padding-bottom: 50px; hyphens: auto; overflow-wrap: break-word; text-rendering: optimizeLegibility; font-kerning: normal;}@media (max-width: 600px) { body { font-size: 0.9em; padding: 12px; } h1 { font-size: 1.8em; }}@media print { html { background-color: white; } body { background-color: transparent; color: black; font-size: 12pt; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3, h4 { page-break-after: avoid; }}p { margin: 1em 0;}a { color: #7eb6ff;}a:visited { color: #b19eff;}img { max-width: 100%;}svg { height: auto; max-width: 100%;}h1, h2, h3, h4, h5, h6 { margin-top: 1.4em; color: #ffffff;}h5, h6 { font-size: 1em; font-style: italic;}h6 { font-weight: normal;}ol, ul { padding-left: 1.7em; margin-top: 1em;}li > ol, li > ul { margin-top: 0;}blockquote { margin: 1em 0 1em 1.7em; padding-left: 1em; border-left: 2px solid #404040; color: #b0b0b0;}code { font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace; font-size: 85%; margin: 0; hyphens: manual; background-color: #2d2d2d; color: #e0e0e0; padding: 0.2em 0.4em; border-radius: 3px;}pre { margin: 1em 0; overflow: auto; background-color: #2d2d2d; padding: 1em; border-radius: 5px;}pre code { padding: 0; overflow: visible; overflow-wrap: normal; background-color: transparent;}.sourceCode { background-color: transparent; overflow: visible;}hr { background-color: #404040; border: none; height: 1px; margin: 1em 0;}table { margin: 1em 0; border-collapse: collapse; width: 100%; overflow-x: auto; display: block; font-variant-numeric: lining-nums tabular-nums;}table caption { margin-bottom: 0.75em;}tbody { margin-top: 0.5em; border-top: 1px solid #404040; border-bottom: 1px solid #404040;}th { border-top: 1px solid #404040; padding: 0.25em 0.5em 0.25em 0.5em;}td { padding: 0.125em 0.5em 0.25em 0.5em;}header { margin-bottom: 4em; text-align: center;}#TOC li { list-style: none;}#TOC ul { padding-left: 1.3em;}#TOC > ul { padding-left: 0;}#TOC a:not(:hover) { text-decoration: none;}code{white-space: pre-wrap;}span.smallcaps{font-variant: small-caps;}div.columns{display: flex; gap: min(4vw, 1.5em);}div.column{flex: auto; overflow-x: auto;}div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}ul.task-list[class]{list-style: none;}ul.task-list li input[type="checkbox"] { font-size: inherit; width: 0.8em; margin: 0 0.8em 0.2em -1.6em; vertical-align: middle;}.display.math{display: block; text-align: center; margin: 0.5rem auto;}</style>Data-Driven 11ty Site Generation
For my 47 Watch project, I edit Executive Orders (et al.) as markdown documents with a particular structure, which makes it easy to turn them into data. After each new edit session in 47 Watch, I rebuild a bunch of data files, including this 11ty master JSON file that contains all the posts nestled up into a nice data frame-esque structure.
While I could have had a script generate dozens (or, eventually hundreds) of 11thy markdown post files I did not have to. All I needed to do was ensure that aforelinked
executive-orders.11tydata.jsonfile was in the right directory and then take advantage of 11ty’s excellent data-driven document generation capability.On
build, 11ty will pair up the base template (in this caseexecutive-order.njk) with that data file, then proceed to generate as many of the files as there are in the array. The index for that collection will also iterate over the array.There are a few Community posts which will help you explore this feature further (or you can clone and play with the 47 Watch code directly).
Pagefind
Pagefind is an open-source library that addresses a common challenge in static site development: providing robust search capabilities without relying on external services or heavy server infrastructure.
I ended up using it for 47 Watch as it has a pretty innovative approach to handling large-scale search operations. The library employs a clever chunking mechanism for its search index, ensuring that folks only download the necessary portions of the index when performing searches. This architectural decision yields some pretty impressive performance metrics — imagine being able to implement full-text search across 10,000 pages while keeping the network payload under 300kB, including the library itself.
The framework-agnostic nature of Pagefind is especially valuable if you’re a gadfly like I am. Whether you’re working with modern frameworks like Next.js and SvelteKit or traditional static site generators like Jekyll and Hugo, Pagefind operates on the final built files, which eliminates the need for framework-specific implementations or affordances.
Pagefind’s also has multilingual support and that plus the rich filtering engine makes it particularly suitable for knowledge bases and research-oriented websites. The ability to customize content weighting and metadata tracking allows for fine-tuned search experiences that can be tailored to specific academic or research needs.
The library’s capability to index non-traditional content types through its NodeJS indexing library — including PDFs and JSON files — extends its utility beyond simple web page searches, making it a versatile tool for comprehensive digital resource management.
If you look at some of the provided demos (MDN, Godot documentation, and XKCD), you can observe for yourself how Pagefind maintains its performance.
All I really had to do was add
"postbuild": "pagefind --site _site",to thepackage.jsonand it worked the first time.I had originally intended to use Algolia, but I’m trying to avoid external third-party dependencies for this project, and Pagefind has all of the features I need for the immediate future.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- 🐘 Mastodon via
@[email protected] - 🦋 Bluesky via
https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy
Also, refer to:
to see how to access a regularly updated database of all the Drops with extracted links, and full-text search capability. ☮️
#1a1a1a #2d2d2d #404040 #7eb6ff #b0b0b0 #b19eff #e0e0e0 #ffffff #TOC
-
Drop #603 (2025-02-12): Suggest • Static • Search
Zed Edit Predictions (Beta); Data-Driven 11ty Site Generation; Pagefind
The Drop is back after the $WORK-fueled hiatus. Travel to and from San Diego was horrendous, and it took most of the weekend to recover from the ~36 hours it took me to get from San Diego back to the Maine compound. The missed Monday Drop was due to porting 47 Watch to Eleventy/11ty. We’ll talk more about that in one of the sections. Tuesday’s miss was due to the ~57 minutes of slumber between Monday and Tuesday thanks to the still unabated long covid resurgence.
Anyway…
Thanks to said 11ty work, today’s Drop showcases some of the tech used to make the new site.
TL;DR
(This is an AI-generated summary of today’s Drop using Ollama + llama 3.2 and a custom prompt.)
- Zed’s new beta edit predictions feature offers multi-line code completion and cross-language awareness, proving especially useful for mixed-language projects and repetitive code modifications.
- 11ty’s data-driven document generation capability enables efficient site building by pairing templates with JSON data files, eliminating the need for multiple individual markdown files.
- Pagefind provides framework-agnostic static site search functionality with efficient chunked indexing, supporting multiple languages and content types while maintaining small network payloads.
Zed Edit Predictions (Beta)
Zed Completions BetaI’ve almost entirely switched to Zed for everything but R work (where I use RStudio, since I will not suffer Positron on my systems due to the Microsoft corruption); and, I do a great deal of work in Zed every day. Said work usually involves switching between many projects, which may be Go, Rust, Shell, or Java|TypeScript-based, plus a healthy dose of HTML, CSS, and Markdown mixed in.
I make heavy use of most of most of Zed’s features, including the AI Assitant Panel and inline AI assistance (which is incredibly handy in the Zed Alacritty terminal, too). I also let Zed collect telemetry, since I work for a startup who uses quite a bit of telemetry in our Visualizer app (so, “fair’s fair”). I mention that last bit as I guess I use Zed just enough for the Zed team to hit me up to test out their beta edit predictions feature. If you’ve used GitHub Copilot or Cody in VS Code/Codium, you’ve experienced edit predictions there. The TLDR for the feature is that Zed’s “Zeta” model has your file and project context at its disposal, keeps track of what you’ve been typing, and tries to pick what it thinks is the likely desired next bit of code or content. The video in the section header does a fine job showing off the functionality.
I got in the beta right before I started the 11ty conversion of 47 Watch, and I cannot fully descibe how handy it was to perform that port. I haven’t been using VS Codium much, of late, so I don’t know if Cody (et al.) has advanced there enough to have a simialr feature set, but — for starters — Zed’s completion works across multiple source code lines (i.e., it can — with oddly great accuracy — see you’re modifying one tag or bit of code, and can surmize you want to make similar changes to similar lines, and it just “does it” after you tab-complete). In a co-mingled project with a Go microservice back end and HTML/JS/CSS front-end, it has yet to get confused by said source diversity, and consistently suggests proper in-language completions for each file type. In fact, it does an uncanny job suggesting front end route code completions when a new back end REST API handler is added.
This new feature saved me hours during the porting process, and I’m writing this section mostly to suggest (heh) you give Zed a go again (if you’re not using it already), and try to get into the beta.
You can disable it entirely or per-project if you don’t want it showing up in, say, a markdown document like the one I’m editing right now, where I want my own words.
I’ll risk one more bit of extended blather and also note that it is super important to treat all the AI “code for me” tools as actual junior-grade assistants, where you sort of trust the output of the task you’ve given them, but also verify and fully grok said output before using it. And, (finally), I also highly suggest — every so often — doing some of the boring coding you may be offloading to LLM assistants. Think of them as katas that keep your brain grounded in the fundamentals. It’s way too easy to let certain skills atropy if you’re relying on a power tool to do the work for you.
Zed Bonus Section
I snuck in this bonus section since one of the few things that kept me from using Zed for editing the Drop posts was the lack of an HTML-based preview for markdown documents. The built-in previewer is fast and looks great, but I can’t copy from it, paste the contents into the WordPress editor, and have formatted content ready to go.
This Zed task snippet:
{ "label": "markdown preview", "command": "pandoc ${ZED_RELATIVE_FILE} -f markdown -t html --metadata title=' ' --no-highlight -s --include-in-header=/Users/hrbrmstr/.cache/darkmeta.html -o /tmp/markdown-preview.html && open /tmp/markdown-preview.html"},Uses pandoc with a dark-mode template (see below) to provide the functionality I’m used to from VS Codium. We’ll eventually talk more about Zed tasks in a future Drop.
<meta name="color-scheme" content="dark light"><style>* { font-family: sans-serif;}html { color: #e0e0e0; background-color: #1a1a1a;}body { margin: 0 auto; max-width: 36em; padding-left: 50px; padding-right: 50px; padding-top: 50px; padding-bottom: 50px; hyphens: auto; overflow-wrap: break-word; text-rendering: optimizeLegibility; font-kerning: normal;}@media (max-width: 600px) { body { font-size: 0.9em; padding: 12px; } h1 { font-size: 1.8em; }}@media print { html { background-color: white; } body { background-color: transparent; color: black; font-size: 12pt; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3, h4 { page-break-after: avoid; }}p { margin: 1em 0;}a { color: #7eb6ff;}a:visited { color: #b19eff;}img { max-width: 100%;}svg { height: auto; max-width: 100%;}h1, h2, h3, h4, h5, h6 { margin-top: 1.4em; color: #ffffff;}h5, h6 { font-size: 1em; font-style: italic;}h6 { font-weight: normal;}ol, ul { padding-left: 1.7em; margin-top: 1em;}li > ol, li > ul { margin-top: 0;}blockquote { margin: 1em 0 1em 1.7em; padding-left: 1em; border-left: 2px solid #404040; color: #b0b0b0;}code { font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace; font-size: 85%; margin: 0; hyphens: manual; background-color: #2d2d2d; color: #e0e0e0; padding: 0.2em 0.4em; border-radius: 3px;}pre { margin: 1em 0; overflow: auto; background-color: #2d2d2d; padding: 1em; border-radius: 5px;}pre code { padding: 0; overflow: visible; overflow-wrap: normal; background-color: transparent;}.sourceCode { background-color: transparent; overflow: visible;}hr { background-color: #404040; border: none; height: 1px; margin: 1em 0;}table { margin: 1em 0; border-collapse: collapse; width: 100%; overflow-x: auto; display: block; font-variant-numeric: lining-nums tabular-nums;}table caption { margin-bottom: 0.75em;}tbody { margin-top: 0.5em; border-top: 1px solid #404040; border-bottom: 1px solid #404040;}th { border-top: 1px solid #404040; padding: 0.25em 0.5em 0.25em 0.5em;}td { padding: 0.125em 0.5em 0.25em 0.5em;}header { margin-bottom: 4em; text-align: center;}#TOC li { list-style: none;}#TOC ul { padding-left: 1.3em;}#TOC > ul { padding-left: 0;}#TOC a:not(:hover) { text-decoration: none;}code{white-space: pre-wrap;}span.smallcaps{font-variant: small-caps;}div.columns{display: flex; gap: min(4vw, 1.5em);}div.column{flex: auto; overflow-x: auto;}div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}ul.task-list[class]{list-style: none;}ul.task-list li input[type="checkbox"] { font-size: inherit; width: 0.8em; margin: 0 0.8em 0.2em -1.6em; vertical-align: middle;}.display.math{display: block; text-align: center; margin: 0.5rem auto;}</style>Data-Driven 11ty Site Generation
For my 47 Watch project, I edit Executive Orders (et al.) as markdown documents with a particular structure, which makes it easy to turn them into data. After each new edit session in 47 Watch, I rebuild a bunch of data files, including this 11ty master JSON file that contains all the posts nestled up into a nice data frame-esque structure.
While I could have had a script generate dozens (or, eventually hundreds) of 11thy markdown post files I did not have to. All I needed to do was ensure that aforelinked
executive-orders.11tydata.jsonfile was in the right directory and then take advantage of 11ty’s excellent data-driven document generation capability.On
build, 11ty will pair up the base template (in this caseexecutive-order.njk) with that data file, then proceed to generate as many of the files as there are in the array. The index for that collection will also iterate over the array.There are a few Community posts which will help you explore this feature further (or you can clone and play with the 47 Watch code directly).
Pagefind
Pagefind is an open-source library that addresses a common challenge in static site development: providing robust search capabilities without relying on external services or heavy server infrastructure.
I ended up using it for 47 Watch as it has a pretty innovative approach to handling large-scale search operations. The library employs a clever chunking mechanism for its search index, ensuring that folks only download the necessary portions of the index when performing searches. This architectural decision yields some pretty impressive performance metrics — imagine being able to implement full-text search across 10,000 pages while keeping the network payload under 300kB, including the library itself.
The framework-agnostic nature of Pagefind is especially valuable if you’re a gadfly like I am. Whether you’re working with modern frameworks like Next.js and SvelteKit or traditional static site generators like Jekyll and Hugo, Pagefind operates on the final built files, which eliminates the need for framework-specific implementations or affordances.
Pagefind’s also has multilingual support and that plus the rich filtering engine makes it particularly suitable for knowledge bases and research-oriented websites. The ability to customize content weighting and metadata tracking allows for fine-tuned search experiences that can be tailored to specific academic or research needs.
The library’s capability to index non-traditional content types through its NodeJS indexing library — including PDFs and JSON files — extends its utility beyond simple web page searches, making it a versatile tool for comprehensive digital resource management.
If you look at some of the provided demos (MDN, Godot documentation, and XKCD), you can observe for yourself how Pagefind maintains its performance.
All I really had to do was add
"postbuild": "pagefind --site _site",to thepackage.jsonand it worked the first time.I had originally intended to use Algolia, but I’m trying to avoid external third-party dependencies for this project, and Pagefind has all of the features I need for the immediate future.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- 🐘 Mastodon via
@[email protected] - 🦋 Bluesky via
https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy
Also, refer to:
to see how to access a regularly updated database of all the Drops with extracted links, and full-text search capability. ☮️
#1a1a1a #2d2d2d #404040 #7eb6ff #b0b0b0 #b19eff #e0e0e0 #ffffff #TOC
-
Drop #603 (2025-02-12): Suggest • Static • Search
Zed Edit Predictions (Beta); Data-Driven 11ty Site Generation; Pagefind
The Drop is back after the $WORK-fueled hiatus. Travel to and from San Diego was horrendous, and it took most of the weekend to recover from the ~36 hours it took me to get from San Diego back to the Maine compound. The missed Monday Drop was due to porting 47 Watch to Eleventy/11ty. We’ll talk more about that in one of the sections. Tuesday’s miss was due to the ~57 minutes of slumber between Monday and Tuesday thanks to the still unabated long covid resurgence.
Anyway…
Thanks to said 11ty work, today’s Drop showcases some of the tech used to make the new site.
TL;DR
(This is an AI-generated summary of today’s Drop using Ollama + llama 3.2 and a custom prompt.)
- Zed’s new beta edit predictions feature offers multi-line code completion and cross-language awareness, proving especially useful for mixed-language projects and repetitive code modifications.
- 11ty’s data-driven document generation capability enables efficient site building by pairing templates with JSON data files, eliminating the need for multiple individual markdown files.
- Pagefind provides framework-agnostic static site search functionality with efficient chunked indexing, supporting multiple languages and content types while maintaining small network payloads.
Zed Edit Predictions (Beta)
Zed Completions BetaI’ve almost entirely switched to Zed for everything but R work (where I use RStudio, since I will not suffer Positron on my systems due to the Microsoft corruption); and, I do a great deal of work in Zed every day. Said work usually involves switching between many projects, which may be Go, Rust, Shell, or Java|TypeScript-based, plus a healthy dose of HTML, CSS, and Markdown mixed in.
I make heavy use of most of most of Zed’s features, including the AI Assitant Panel and inline AI assistance (which is incredibly handy in the Zed Alacritty terminal, too). I also let Zed collect telemetry, since I work for a startup who uses quite a bit of telemetry in our Visualizer app (so, “fair’s fair”). I mention that last bit as I guess I use Zed just enough for the Zed team to hit me up to test out their beta edit predictions feature. If you’ve used GitHub Copilot or Cody in VS Code/Codium, you’ve experienced edit predictions there. The TLDR for the feature is that Zed’s “Zeta” model has your file and project context at its disposal, keeps track of what you’ve been typing, and tries to pick what it thinks is the likely desired next bit of code or content. The video in the section header does a fine job showing off the functionality.
I got in the beta right before I started the 11ty conversion of 47 Watch, and I cannot fully descibe how handy it was to perform that port. I haven’t been using VS Codium much, of late, so I don’t know if Cody (et al.) has advanced there enough to have a simialr feature set, but — for starters — Zed’s completion works across multiple source code lines (i.e., it can — with oddly great accuracy — see you’re modifying one tag or bit of code, and can surmize you want to make similar changes to similar lines, and it just “does it” after you tab-complete). In a co-mingled project with a Go microservice back end and HTML/JS/CSS front-end, it has yet to get confused by said source diversity, and consistently suggests proper in-language completions for each file type. In fact, it does an uncanny job suggesting front end route code completions when a new back end REST API handler is added.
This new feature saved me hours during the porting process, and I’m writing this section mostly to suggest (heh) you give Zed a go again (if you’re not using it already), and try to get into the beta.
You can disable it entirely or per-project if you don’t want it showing up in, say, a markdown document like the one I’m editing right now, where I want my own words.
I’ll risk one more bit of extended blather and also note that it is super important to treat all the AI “code for me” tools as actual junior-grade assistants, where you sort of trust the output of the task you’ve given them, but also verify and fully grok said output before using it. And, (finally), I also highly suggest — every so often — doing some of the boring coding you may be offloading to LLM assistants. Think of them as katas that keep your brain grounded in the fundamentals. It’s way too easy to let certain skills atropy if you’re relying on a power tool to do the work for you.
Zed Bonus Section
I snuck in this bonus section since one of the few things that kept me from using Zed for editing the Drop posts was the lack of an HTML-based preview for markdown documents. The built-in previewer is fast and looks great, but I can’t copy from it, paste the contents into the WordPress editor, and have formatted content ready to go.
This Zed task snippet:
{ "label": "markdown preview", "command": "pandoc ${ZED_RELATIVE_FILE} -f markdown -t html --metadata title=' ' --no-highlight -s --include-in-header=/Users/hrbrmstr/.cache/darkmeta.html -o /tmp/markdown-preview.html && open /tmp/markdown-preview.html"},Uses pandoc with a dark-mode template (see below) to provide the functionality I’m used to from VS Codium. We’ll eventually talk more about Zed tasks in a future Drop.
<meta name="color-scheme" content="dark light"><style>* { font-family: sans-serif;}html { color: #e0e0e0; background-color: #1a1a1a;}body { margin: 0 auto; max-width: 36em; padding-left: 50px; padding-right: 50px; padding-top: 50px; padding-bottom: 50px; hyphens: auto; overflow-wrap: break-word; text-rendering: optimizeLegibility; font-kerning: normal;}@media (max-width: 600px) { body { font-size: 0.9em; padding: 12px; } h1 { font-size: 1.8em; }}@media print { html { background-color: white; } body { background-color: transparent; color: black; font-size: 12pt; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3, h4 { page-break-after: avoid; }}p { margin: 1em 0;}a { color: #7eb6ff;}a:visited { color: #b19eff;}img { max-width: 100%;}svg { height: auto; max-width: 100%;}h1, h2, h3, h4, h5, h6 { margin-top: 1.4em; color: #ffffff;}h5, h6 { font-size: 1em; font-style: italic;}h6 { font-weight: normal;}ol, ul { padding-left: 1.7em; margin-top: 1em;}li > ol, li > ul { margin-top: 0;}blockquote { margin: 1em 0 1em 1.7em; padding-left: 1em; border-left: 2px solid #404040; color: #b0b0b0;}code { font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace; font-size: 85%; margin: 0; hyphens: manual; background-color: #2d2d2d; color: #e0e0e0; padding: 0.2em 0.4em; border-radius: 3px;}pre { margin: 1em 0; overflow: auto; background-color: #2d2d2d; padding: 1em; border-radius: 5px;}pre code { padding: 0; overflow: visible; overflow-wrap: normal; background-color: transparent;}.sourceCode { background-color: transparent; overflow: visible;}hr { background-color: #404040; border: none; height: 1px; margin: 1em 0;}table { margin: 1em 0; border-collapse: collapse; width: 100%; overflow-x: auto; display: block; font-variant-numeric: lining-nums tabular-nums;}table caption { margin-bottom: 0.75em;}tbody { margin-top: 0.5em; border-top: 1px solid #404040; border-bottom: 1px solid #404040;}th { border-top: 1px solid #404040; padding: 0.25em 0.5em 0.25em 0.5em;}td { padding: 0.125em 0.5em 0.25em 0.5em;}header { margin-bottom: 4em; text-align: center;}#TOC li { list-style: none;}#TOC ul { padding-left: 1.3em;}#TOC > ul { padding-left: 0;}#TOC a:not(:hover) { text-decoration: none;}code{white-space: pre-wrap;}span.smallcaps{font-variant: small-caps;}div.columns{display: flex; gap: min(4vw, 1.5em);}div.column{flex: auto; overflow-x: auto;}div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}ul.task-list[class]{list-style: none;}ul.task-list li input[type="checkbox"] { font-size: inherit; width: 0.8em; margin: 0 0.8em 0.2em -1.6em; vertical-align: middle;}.display.math{display: block; text-align: center; margin: 0.5rem auto;}</style>Data-Driven 11ty Site Generation
For my 47 Watch project, I edit Executive Orders (et al.) as markdown documents with a particular structure, which makes it easy to turn them into data. After each new edit session in 47 Watch, I rebuild a bunch of data files, including this 11ty master JSON file that contains all the posts nestled up into a nice data frame-esque structure.
While I could have had a script generate dozens (or, eventually hundreds) of 11thy markdown post files I did not have to. All I needed to do was ensure that aforelinked
executive-orders.11tydata.jsonfile was in the right directory and then take advantage of 11ty’s excellent data-driven document generation capability.On
build, 11ty will pair up the base template (in this caseexecutive-order.njk) with that data file, then proceed to generate as many of the files as there are in the array. The index for that collection will also iterate over the array.There are a few Community posts which will help you explore this feature further (or you can clone and play with the 47 Watch code directly).
Pagefind
Pagefind is an open-source library that addresses a common challenge in static site development: providing robust search capabilities without relying on external services or heavy server infrastructure.
I ended up using it for 47 Watch as it has a pretty innovative approach to handling large-scale search operations. The library employs a clever chunking mechanism for its search index, ensuring that folks only download the necessary portions of the index when performing searches. This architectural decision yields some pretty impressive performance metrics — imagine being able to implement full-text search across 10,000 pages while keeping the network payload under 300kB, including the library itself.
The framework-agnostic nature of Pagefind is especially valuable if you’re a gadfly like I am. Whether you’re working with modern frameworks like Next.js and SvelteKit or traditional static site generators like Jekyll and Hugo, Pagefind operates on the final built files, which eliminates the need for framework-specific implementations or affordances.
Pagefind’s also has multilingual support and that plus the rich filtering engine makes it particularly suitable for knowledge bases and research-oriented websites. The ability to customize content weighting and metadata tracking allows for fine-tuned search experiences that can be tailored to specific academic or research needs.
The library’s capability to index non-traditional content types through its NodeJS indexing library — including PDFs and JSON files — extends its utility beyond simple web page searches, making it a versatile tool for comprehensive digital resource management.
If you look at some of the provided demos (MDN, Godot documentation, and XKCD), you can observe for yourself how Pagefind maintains its performance.
All I really had to do was add
"postbuild": "pagefind --site _site",to thepackage.jsonand it worked the first time.I had originally intended to use Algolia, but I’m trying to avoid external third-party dependencies for this project, and Pagefind has all of the features I need for the immediate future.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- 🐘 Mastodon via
@[email protected] - 🦋 Bluesky via
https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy
Also, refer to:
to see how to access a regularly updated database of all the Drops with extracted links, and full-text search capability. ☮️
#1a1a1a #2d2d2d #404040 #7eb6ff #b0b0b0 #b19eff #e0e0e0 #ffffff #TOC
-
Drop #603 (2025-02-12): Suggest • Static • Search
Zed Edit Predictions (Beta); Data-Driven 11ty Site Generation; Pagefind
The Drop is back after the $WORK-fueled hiatus. Travel to and from San Diego was horrendous, and it took most of the weekend to recover from the ~36 hours it took me to get from San Diego back to the Maine compound. The missed Monday Drop was due to porting 47 Watch to Eleventy/11ty. We’ll talk more about that in one of the sections. Tuesday’s miss was due to the ~57 minutes of slumber between Monday and Tuesday thanks to the still unabated long covid resurgence.
Anyway…
Thanks to said 11ty work, today’s Drop showcases some of the tech used to make the new site.
TL;DR
(This is an AI-generated summary of today’s Drop using Ollama + llama 3.2 and a custom prompt.)
- Zed’s new beta edit predictions feature offers multi-line code completion and cross-language awareness, proving especially useful for mixed-language projects and repetitive code modifications.
- 11ty’s data-driven document generation capability enables efficient site building by pairing templates with JSON data files, eliminating the need for multiple individual markdown files.
- Pagefind provides framework-agnostic static site search functionality with efficient chunked indexing, supporting multiple languages and content types while maintaining small network payloads.
Zed Edit Predictions (Beta)
Zed Completions BetaI’ve almost entirely switched to Zed for everything but R work (where I use RStudio, since I will not suffer Positron on my systems due to the Microsoft corruption); and, I do a great deal of work in Zed every day. Said work usually involves switching between many projects, which may be Go, Rust, Shell, or Java|TypeScript-based, plus a healthy dose of HTML, CSS, and Markdown mixed in.
I make heavy use of most of most of Zed’s features, including the AI Assitant Panel and inline AI assistance (which is incredibly handy in the Zed Alacritty terminal, too). I also let Zed collect telemetry, since I work for a startup who uses quite a bit of telemetry in our Visualizer app (so, “fair’s fair”). I mention that last bit as I guess I use Zed just enough for the Zed team to hit me up to test out their beta edit predictions feature. If you’ve used GitHub Copilot or Cody in VS Code/Codium, you’ve experienced edit predictions there. The TLDR for the feature is that Zed’s “Zeta” model has your file and project context at its disposal, keeps track of what you’ve been typing, and tries to pick what it thinks is the likely desired next bit of code or content. The video in the section header does a fine job showing off the functionality.
I got in the beta right before I started the 11ty conversion of 47 Watch, and I cannot fully descibe how handy it was to perform that port. I haven’t been using VS Codium much, of late, so I don’t know if Cody (et al.) has advanced there enough to have a simialr feature set, but — for starters — Zed’s completion works across multiple source code lines (i.e., it can — with oddly great accuracy — see you’re modifying one tag or bit of code, and can surmize you want to make similar changes to similar lines, and it just “does it” after you tab-complete). In a co-mingled project with a Go microservice back end and HTML/JS/CSS front-end, it has yet to get confused by said source diversity, and consistently suggests proper in-language completions for each file type. In fact, it does an uncanny job suggesting front end route code completions when a new back end REST API handler is added.
This new feature saved me hours during the porting process, and I’m writing this section mostly to suggest (heh) you give Zed a go again (if you’re not using it already), and try to get into the beta.
You can disable it entirely or per-project if you don’t want it showing up in, say, a markdown document like the one I’m editing right now, where I want my own words.
I’ll risk one more bit of extended blather and also note that it is super important to treat all the AI “code for me” tools as actual junior-grade assistants, where you sort of trust the output of the task you’ve given them, but also verify and fully grok said output before using it. And, (finally), I also highly suggest — every so often — doing some of the boring coding you may be offloading to LLM assistants. Think of them as katas that keep your brain grounded in the fundamentals. It’s way too easy to let certain skills atropy if you’re relying on a power tool to do the work for you.
Zed Bonus Section
I snuck in this bonus section since one of the few things that kept me from using Zed for editing the Drop posts was the lack of an HTML-based preview for markdown documents. The built-in previewer is fast and looks great, but I can’t copy from it, paste the contents into the WordPress editor, and have formatted content ready to go.
This Zed task snippet:
{ "label": "markdown preview", "command": "pandoc ${ZED_RELATIVE_FILE} -f markdown -t html --metadata title=' ' --no-highlight -s --include-in-header=/Users/hrbrmstr/.cache/darkmeta.html -o /tmp/markdown-preview.html && open /tmp/markdown-preview.html"},Uses pandoc with a dark-mode template (see below) to provide the functionality I’m used to from VS Codium. We’ll eventually talk more about Zed tasks in a future Drop.
<meta name="color-scheme" content="dark light"><style>* { font-family: sans-serif;}html { color: #e0e0e0; background-color: #1a1a1a;}body { margin: 0 auto; max-width: 36em; padding-left: 50px; padding-right: 50px; padding-top: 50px; padding-bottom: 50px; hyphens: auto; overflow-wrap: break-word; text-rendering: optimizeLegibility; font-kerning: normal;}@media (max-width: 600px) { body { font-size: 0.9em; padding: 12px; } h1 { font-size: 1.8em; }}@media print { html { background-color: white; } body { background-color: transparent; color: black; font-size: 12pt; } p, h2, h3 { orphans: 3; widows: 3; } h2, h3, h4 { page-break-after: avoid; }}p { margin: 1em 0;}a { color: #7eb6ff;}a:visited { color: #b19eff;}img { max-width: 100%;}svg { height: auto; max-width: 100%;}h1, h2, h3, h4, h5, h6 { margin-top: 1.4em; color: #ffffff;}h5, h6 { font-size: 1em; font-style: italic;}h6 { font-weight: normal;}ol, ul { padding-left: 1.7em; margin-top: 1em;}li > ol, li > ul { margin-top: 0;}blockquote { margin: 1em 0 1em 1.7em; padding-left: 1em; border-left: 2px solid #404040; color: #b0b0b0;}code { font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace; font-size: 85%; margin: 0; hyphens: manual; background-color: #2d2d2d; color: #e0e0e0; padding: 0.2em 0.4em; border-radius: 3px;}pre { margin: 1em 0; overflow: auto; background-color: #2d2d2d; padding: 1em; border-radius: 5px;}pre code { padding: 0; overflow: visible; overflow-wrap: normal; background-color: transparent;}.sourceCode { background-color: transparent; overflow: visible;}hr { background-color: #404040; border: none; height: 1px; margin: 1em 0;}table { margin: 1em 0; border-collapse: collapse; width: 100%; overflow-x: auto; display: block; font-variant-numeric: lining-nums tabular-nums;}table caption { margin-bottom: 0.75em;}tbody { margin-top: 0.5em; border-top: 1px solid #404040; border-bottom: 1px solid #404040;}th { border-top: 1px solid #404040; padding: 0.25em 0.5em 0.25em 0.5em;}td { padding: 0.125em 0.5em 0.25em 0.5em;}header { margin-bottom: 4em; text-align: center;}#TOC li { list-style: none;}#TOC ul { padding-left: 1.3em;}#TOC > ul { padding-left: 0;}#TOC a:not(:hover) { text-decoration: none;}code{white-space: pre-wrap;}span.smallcaps{font-variant: small-caps;}div.columns{display: flex; gap: min(4vw, 1.5em);}div.column{flex: auto; overflow-x: auto;}div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}ul.task-list[class]{list-style: none;}ul.task-list li input[type="checkbox"] { font-size: inherit; width: 0.8em; margin: 0 0.8em 0.2em -1.6em; vertical-align: middle;}.display.math{display: block; text-align: center; margin: 0.5rem auto;}</style>Data-Driven 11ty Site Generation
For my 47 Watch project, I edit Executive Orders (et al.) as markdown documents with a particular structure, which makes it easy to turn them into data. After each new edit session in 47 Watch, I rebuild a bunch of data files, including this 11ty master JSON file that contains all the posts nestled up into a nice data frame-esque structure.
While I could have had a script generate dozens (or, eventually hundreds) of 11thy markdown post files I did not have to. All I needed to do was ensure that aforelinked
executive-orders.11tydata.jsonfile was in the right directory and then take advantage of 11ty’s excellent data-driven document generation capability.On
build, 11ty will pair up the base template (in this caseexecutive-order.njk) with that data file, then proceed to generate as many of the files as there are in the array. The index for that collection will also iterate over the array.There are a few Community posts which will help you explore this feature further (or you can clone and play with the 47 Watch code directly).
Pagefind
Pagefind is an open-source library that addresses a common challenge in static site development: providing robust search capabilities without relying on external services or heavy server infrastructure.
I ended up using it for 47 Watch as it has a pretty innovative approach to handling large-scale search operations. The library employs a clever chunking mechanism for its search index, ensuring that folks only download the necessary portions of the index when performing searches. This architectural decision yields some pretty impressive performance metrics — imagine being able to implement full-text search across 10,000 pages while keeping the network payload under 300kB, including the library itself.
The framework-agnostic nature of Pagefind is especially valuable if you’re a gadfly like I am. Whether you’re working with modern frameworks like Next.js and SvelteKit or traditional static site generators like Jekyll and Hugo, Pagefind operates on the final built files, which eliminates the need for framework-specific implementations or affordances.
Pagefind’s also has multilingual support and that plus the rich filtering engine makes it particularly suitable for knowledge bases and research-oriented websites. The ability to customize content weighting and metadata tracking allows for fine-tuned search experiences that can be tailored to specific academic or research needs.
The library’s capability to index non-traditional content types through its NodeJS indexing library — including PDFs and JSON files — extends its utility beyond simple web page searches, making it a versatile tool for comprehensive digital resource management.
If you look at some of the provided demos (MDN, Godot documentation, and XKCD), you can observe for yourself how Pagefind maintains its performance.
All I really had to do was add
"postbuild": "pagefind --site _site",to thepackage.jsonand it worked the first time.I had originally intended to use Algolia, but I’m trying to avoid external third-party dependencies for this project, and Pagefind has all of the features I need for the immediate future.
FIN
Remember, you can follow and interact with the full text of The Daily Drop’s free posts on:
- 🐘 Mastodon via
@[email protected] - 🦋 Bluesky via
https://bsky.app/profile/dailydrop.hrbrmstr.dev.web.brid.gy
Also, refer to:
to see how to access a regularly updated database of all the Drops with extracted links, and full-text search capability. ☮️
#1a1a1a #2d2d2d #404040 #7eb6ff #b0b0b0 #b19eff #e0e0e0 #ffffff #TOC
-
What is Correlation?
Correlation indicates that as one variable changes in value, the other variable tends to change in a specific direction. For example, the height and weight of an individual can be correlated – which means if the person’s height is on the taller side of the curve, the weight would also be high.
Table of Contents
- What is Correlation?
- What are correlation coefficients?
- Pearson’s correlation coefficient
- Correlation doesn’t imply causation.
What are correlation coefficients?
The correlation coefficient is a value that can quantitatively assess the strength and direction of the correlation or association between the two variables. One can choose different types of correlation coefficients depending on the type of data and the relationship you are looking at. One common correlation coefficient is the Pearson correlation coefficient, which explores the linear relationship between two continuous variables.
Pearson’s correlation coefficient
Before we proceed further, we need to learn how to interpret the correlation coefficient. Look at this by studying Pearson’s correlation coefficient (r)under the following subheadings.
- Strength: The greater the absolute value of the correlation coefficient, the stronger the relationship between the variables. For example, -1 and 1 are the extremes in the range, indicating a linear relationship between the negative and positive variables, respectively. A coefficient of 0 indicates no correlation between the variables.
- Direction: the sign of the correlation coefficient represents the direction of the relationship. For example, a negative sign indicates that the other variable’s value decreases as one variable’s value increases.
- One example of a negative correlation would be the engine’s horsepower and mileage per litre of gas or petrol. Engines with higher horsepower tend to consume more fuel, thus resulting in lower mileage per litre of gas or petrol.
Scatterplots are one of the best graphs to visualise and look for correlations between the variables. Here is an image of scatterplots featuring different Pearson’s correlation coefficients: -1, -0.5, 0, 0.5, 1
As you can see from the above graph, the stronger the correlation, the closer the data points are to the line, and the less the dispersion of the data points on the graph.
Pearson’s correlation coefficient measures only the linear relationship betwene the variables
If there is a curvilinear relationship between the variables, the person’s correlation coefficient might not detect it and might give you wrong, inaccurate values.
Correlation doesn’t imply causation.
You might have heard this rather infamous quote before: “Correlation doesn’t imply causation”. This is an important line to remember. Correlation doesn’t mean that changes in one variable will lead to changes in the value of the other variable.
- So, if two variables are correlated, but there is no causal relationship, how can one explain the correlation between the two variables?
- A third variable might correlate with the other two variables, leading to the correlation between the first two variables. This third variable is called a confounder or confounding variable. This confounding variable correlates with the other two variables and creates confusion regarding which is a causal relationship and which is a spurious association. In statistics and trials, you need to perform a randomized controlled trial
Like this post? Share and subscribe to stay in the loop: Subscribe
-
Import JetPack Statistics into Koko
https://shkspr.mobi/blog/2024/10/import-jetpack-statistics-into-koko/I've quit JetPack stats. I've moved to Koko Analytics. All the stats code is self hosted, it is privacy preserving, and the codebase is small and simple.
But I am vain. I want all my old JetPack stats to appear in Koko so I can look back on the glory days of blogging.
Koko has two main tables. The first is a summary table called
datevisitorspageviews2009-10-3025472009-10-3170862009-11-016172wpbp_koko_analytics_site_stats:That's the total number of visitors and page views for each date.
Then there's a more detailed breakdown at
dateidvisitorspageviews2009-10-30123222009-10-30456562009-10-3078911wpbp_koko_analytics_post_statsThat shows every individual post's number of views and visitors per day.
WordPress's database is MySQL. It can handle CSV imports. So, given these tables are pretty simple, it is possible to get the old WordPress stats, convert them to CSV, and then import them.
Let's go!
As described in a previous post, the JetPack stats API is fairly basic. It doesn't differentiate between visitors and pageviews. So, for this import, we'll pretend they're the same.
This will be a 4 step process.
1. Get all your stats in JSON format
This code loops through your stats and downloads a JSON file for each one.
You will need:
- Your API key - find it at https://apikey.wordpress.com/
- Your blog's web address - I assume you know this
- The earliest date you have for JetPack - you will need to find this yourself
import requestsimport datetimeimport osimport json# Directory to save the JSON filessave_dir = "jetpack_stats"os.makedirs(save_dir, exist_ok=True)# URL of the APIbase_url = "https://stats.wordpress.com/csv.php?api_key=123456789012"+\ "&blog_uri=https://example.com/"+\ "&table=postviews"+\ "&days=1"+\ "&format=json"+\ "&limit=-1"+\ "&end="# Make API call and save the responsedef fetch_and_save_json(date): # Format the date as ISO8601 (YYYY-MM-DD) formatted_date = date.isoformat() # Make the API call url = f"{base_url}{formatted_date}" response = requests.get(url) if response.status_code == 200: data = response.json() file_name = f"{formatted_date}.json" file_path = os.path.join(save_dir, file_name) with open(file_path, "w") as f: json.dump(data, f, indent=4) print(f"Saved {formatted_date}") else: print(f"Failed! {formatted_date} status code: {response.status_code}")# Iterate over a date rangestart_date = datetime.date(2020, 1 , 1)end_date = datetime.date(2024, 10, 30)# Loop through all dates current_date = start_datewhile current_date <= end_date: fetch_and_save_json(current_date) current_date += datetime.timedelta(days=1)2. Generate The Individual Posts' Stats
This takes those JSON files and creates a single CSV ready to upload to
wpbp_koko_analytics_post_stats.import osimport jsonimport csv# Directory where the JSON files are storedjson_dir = "jetpack_stats"# List to hold the loaded dataall_data = []json_files = sorted( [ file for file in os.listdir( json_dir ) if file.endswith( ".json" ) ] )# Loop through all files in the directoryfor file_name in json_files : if file_name.endswith( ".json" ) : file_path = os.path.join(json_dir, file_name) with open(file_path, "r") as json_file : data = json.load(json_file) all_data.append(data) # Add the data to the listprint(f"Total files loaded: {len(all_data)}")with open( "wpbp_koko_analytics_post_stats.csv", "w", newline="") as csvfile: csvwriter = csv.writer( csvfile, delimiter="," ) for stat in all_data: for views in stat[0]["postviews"] : csvwriter.writerow( [ stat[0]["date"], views["post_id"], views["views"], views["views"] ] )3. Generate the total site views
It is possible to get this separately from the JetPack API using
&table=views- but that's a lot more API calls. So we're just going to sum it up instead 😄This, again, inserts a dummy value for visitors.
import pandas as pdinput_csv = "wpbp_koko_analytics_post_stats.csv"output_csv = "wpbp_koko_analytics_site_stats.csv"column_names = ['Date', 'Post ID', 'Visitors', 'Page Views']df = pd.read_csv( input_csv, names=column_names )# Group by Date and sum the Page Viewsdf_grouped = df.groupby( "Date" )["Page Views"].sum().reset_index()# Add a new column with a copy of the Page Viewsdf_grouped['Visitors'] = df_grouped['Page Views']# Rename the Page Views column to Total Page Viewsdf_grouped = df_grouped.rename(columns={"Page Views": "Total Page Views"})# Write the dataframe to the output CSV filedf_grouped.to_csv( output_csv, index=False, header=False )4. Upload to MySQL
You're on your own here, Sparky. If you have something like PHPMyAdmin, you should be able to load the file in directly. Anything else… good luck!
Once done, your stats dashboard should be filled with historic data.
Huge thanks to Koko Analytics for providing such a great tool and answering my questions.
https://shkspr.mobi/blog/2024/10/import-jetpack-statistics-into-koko/
-
Z80 and AY-3-8910 – Part 2
I’ve spent a bit of time looking at the “Tester” part of the AY driver code for Tim Follin’s music archive that I talked about in Z80 and AY-3-8910.
This is documenting what I think I’ve worked out so far for the tester code.
The Sound Tester
As previously mentioned, there are essentially three parts to the code in Follin archive:
- The tune and effect data.
- Ste Ruddy’s Sound Driver.
- A tracker-style (ish) tester UI application.
The first part looked at the sound driver itself, and essentially skipped over the tester part of the code. This post picks up on that tester code.
Reminder, from part one, the main structure is as follows:
Code_Start: EQU 40000
Data_Start: EQU 50000
;-----------------------------
ORG Code_Start
; The UI/tester code
TESTER:
LOOP: Calls the following for each scan:
HALT - Suspends until an interrupt comes in?
CALL UPDATE
CALL REFRESH
CALL CLOCK
CALL KEYSCAN
Repeat as necessary
KEYSCAN: UI scanning
CLOCK: Possibly maintain a 50Hz refresh rate clock?
UPDATE: Loads the internal state of all sound variables from
the driver and displays them in real time via the UI.
; The sound driver
CODE_TOP:
TUNE: Select which tune to play.
TUNE_IN: Init all internal sound state variables for a new tune.
TUNEOFF: Stop a playing tune, eg to change tune or start an FX.
FX: Start playing an FX.
FLOOP: Keep processing FX instructions until complete.
REFRESH: "run" a scan of the sound driver updating and outputting the soundThe Tester Code
Initialisation information and main screen data:
;**************************************
; Z80 AY MUSIC DRIVER
;**************************************
; ORG 40000
; LOAD 0C000H
;======================================
;STACK DEPTHS
SD: EQU 3
;======================================
ASCII: EQU 23560 ; 23560 = $5C08 = System Variable "LAST K"
TESTER: PUSH AF
PUSH BC
PUSH DE
PUSH HL
XOR A ; ASCII = MINS = SECS = 0
LD (ASCII),A
LD (MINS),A
LD (SECS),A
CALL TUNEOFF ; TUNE initialisation
CALL STACKMESS ; Kick off the Tester code!
DB CLS ; The start of the main UI data
DB AT,0,0
DB INK,01010111B
DB "'AY' MUSIC DRIVE"
DB "R V2 BY S.RUDDY"
... Skip ...
DB INK,64+5
DB "VOLUME "
DB " "
DB 255
... Skip ...
AT: EQU 22
INK: EQU 16
CLS: EQU 15
STACKMESS: POP IX
CALL MESS
JP (IX)There is a whole lot of screen data in DB blocks which includes some “op codes” that are defined later: AT, INK, CLS. These are special codes that are used by the ROM-based print routines (more here), as used by Sinclair BASIC, but in this case they are spelt out directly, later in code. The final 255 signifies the end of the screen data.
So how are these definitions handled? That all comes up in the “MESS” routine I’ll get to in a moment, but first that “STACKMESS” routine needs a bit of explanation.
When a CALL instruction happens, such as the CALL STACKMESS at the start, the current program counter gets pushed onto the stack. In this case the current PC will point to the instruction after the CALL, which happens to be the start of the screen data. So the POP IX will grab the address of the screen data and drop it into IX and then call the “MESS” function to actually get on with it!
But before I get to that, there is some more code after the screen data:
LD HL,CALC1
PUSH HL
LD A,H
LD DE,4067H ; Output high byte
CALL HEX
POP HL
LD A,L
LD DE,4069H ; Output low byte
CALL HEX
LD HL,(CALC2)
PUSH HL
LD A,H
LD DE,4071H ; Output number of Tunes
CALL HEX
POP HL
LD A,L
LD DE,4073H ; Output number of effects
CALL HEX
LD HL,CALC1
LD DE,(CALC2)
ADD HL,DE
PUSH HL
LD A,H
LD DE,407CH ; Not entirely sure what this is outputting...
CALL HEX
POP HL
LD A,L
LD DE,407EH
CALL HEXThis is writing some basic data out to the display. CALC1 seems to relate to code section size. I believe CALC2 is the start address of the tune data, which is the following:
ORG Data_Start
TUNES: EQU 5
EFFECTS: EQU 21All three of these sections are outputting a 16-bit value in two single-byte chunks using the “HEX” routine, which takes a screen address (in the range $4000-$57FF) and outputs a hex number at that screen location.
So while I’m at it then, how is that HEX function working?
;--------------------------------------
HEX: INC DE ; DE contains the screen address to use
PUSH AF ; Start with DE+1
CALL ONEnib ; Write out the LOW 4-bits
POP AF
RRA ; A = A>>4
RRA ; to write out HIGH 4-bits
RRA
RRA
DEC DE ; Back to original DE screen address
ONEnib: AND 15 ; A = A & 0xF
ADD A ; BC = A * 2
LD C,A
LD B,0
LD HL,ROM_TAB ; Read from ROM_TAB[BC]
ADD HL,BC
LD A,(HL)
INC HL
LD H,(HL)
LD L,A ; HL = (uint16_t)ROM_TAB[A]
MIKESbug: LD C,D ; So HL now points to character bitmap in ROM
LD B,8 ; Write out 8 bytes to display memory directly
PRloop: LD A,(HL) ; (DE) = (HL)
LD (DE),A
INC HL ; HL++
INC D ; NB: Layout of display mem means D++ is next line of char
; for same value of E.
DJNZ PRloop ; WHILE (B-- > 0)
LD D,C ; (Restore D before returning, so DE still = screen addr)
RET
ROM_TAB: DW 3D80H ; ROM character set: 3D80 = "0"
DW 3D88H ; Each char = 8 x 8 bits
DW 3D90H
DW 3D98H
DW 3DA0H
DW 3DA8H
DW 3DB0H
DW 3DB8H
DW 3DC0H
DW 3DC8H ; = "9"
DW 3E08H ; = "A"
DW 3E10H
DW 3E18H
DW 3E20H
DW 3E28H
DW 3E30H ; = "F"This is making use of the character set stored in the Spectrum ROM (more here) which is indexed via a 16-word jump table mapping the characters onto each of the 16 hex characters: 0..9, A..F.
Then each byte, 8 in total, of the character is written directly out to the Spectrum screen memory taking advantage of the odd formatting of the screen memory to easily skip to the next line of the display for each line of the character (more here).
So before I get into the main update loop, how the screen initialised and set up? That happens in the “MESS” and some ancillary functions.
MESS: LD A,(IX+0) ; At this point, McursorX, McursorY = (0,0)
INC IX ; So read a byte of screen data
OR A
RET M ; Stop IF A=255 (i.e. negative)
CP 32
JR C,Mcontrol ; IF A<32 process control character then RET back to "MESS"
CALL Mgetchar ; ELSE Process character
CALL Mgetaddr ; Get screen address for next output in DE
CALL MIKESbug ; Output the character
CALL PRattr ; Set the colour attributes
CALL INCcursor ; Update the screen position for the next byte of screen data
JR MESS
Mcontrol: LD HL,MESS ; Stick the address of "MESS" on the stack for the RET
PUSH HL
CP 15 ; IF A == CLS
JR Z,Mcls
CP 22 ; IF A == AT
JP Z,Mat
CP 16 ; IF A == INK
JR Z,Mink
RET ; RETurn to "MESS"
Mcolour: DB 0 ; Working variables for cursor position and colour
McursorX: DB 0
McursorY: DB 0 ; Has to be directly after McursorX (see later)
Mink: LD A,(IX+0) ; Process INK to set colour
INC IX
LD (Mcolour),A
RET
Mcls: LD HL,4000H ; Process CLS to clear screen
LD (HL),L
LD DE,4001H
LD BC,1AFFH
LDIR
LD (McursorX),BC
RET
INCcursor: LD HL,McursorX ; Moves the cursor on one position
LD A,(HL)
INC A
AND 31
LD (HL),A ; X++; X = X % 32
RET NZ ; IF X==0; Y++
INC HL ; Assumes McursorY is McursorX++
INC (HL)
RET
Mgetchar: LD L,A ; HL = A*8 + 3C00
LD H,0 ; Note: A > 32; where 32="Space"
ADD HL,HL ; In ROM, space is address 3D00
ADD HL,HL ; 32 * 8 = 0x100
ADD HL,HL
LD BC,3C00H
ADD HL,BC ; HL = Start address of character map for char in A in ROM
RET
.... skip ....
Mgetaddr: LD A,(McursorY) ; Calculate the screen address for (McursorX, McursorY)
AND 18H
OR 40H
LD D,A
LD A,(McursorY)
RRCA
RRCA
RRCA
AND 0E0H
LD E,A
LD A,(McursorX)
ADD E
LD E,A
RET ; DE = required screen address
Mat: LD A,(IX+0) ; Set cursor to provided X, Y in screen data
LD (McursorX),A
INC IX
LD A,(IX+0)
LD (McursorY),A
INC IX
RET
PRattr: LD A,D ; Get address of ATTRibute memory
RRA
RRA
RRA
AND 3
OR 58H
LD D,A
LD A,(Mcolour)
LD (DE),A ; And set the colour
RETBasically this loop keeps working on the provided screen data until the value 255 is found, at which point it returns. There are two paths for handling the data:
- IF the value is < 32 then it is a control value. Only CLS, AT and INK are recognised.
- ELSE the value is assumed to be an ASCII character and is displayed.
Whatever is happening, happens at the coordinates given by (McursorX, McursorY) which start out as (0,0) and get updated automatically when a character is output, or in response to an AT command. INK will set the required colour in Mcolour, which again starts out as 0. This is applied after the character is written to the screen, using the PRattr function.
There is a fun bit of optimisation going on in Mcontrol. At the start it pushes the address of the MESS function on the stack, which means that the RET will jump back to the start of MESS rather than where the jump happened to Mcontrol itself.
There is another shortcut in the Mcls function: LDIR. From http://z80-heaven.wikidot.com/instructions-set:ldir: “Repeats LDI (LD (DE),(HL), then increments DE, HL, and decrements BC) until BC=0.” By setting the contents of HL (the first byte of the display) to zero, this will tile that same value across the display memory until BC, which starts at $1AFF, is zero. This will zero the whole display – both pixels and attributes – from 0x4000 through to 0x5AFF.
Now finally, we get to the main update loop.
LOOP:
HALT
CALL UPDATE ; Update the display from the current Sound parameters
LD A,2
OUT (254),A ; Set border to 2
CALL REFRESH ; Update the sound driver parameters
XOR A
OUT (254),A ; Set border to 0
CALL CLOCK ; Run 50Hz clock
CALL KEYSCAN ; Guess what - scans the keyboard 🙂
LD A,07FH
IN A,(254) ; Reads 0x7FFE which is the bottom row of the keyboard
AND 1
JP NZ,LOOP ; Checks bit 0, which is the SPACE key
LD BC,65533 ; AY OUTPUT PORTS (FFFD, BFFD)
LD A,7
OUT (C),A
LD BC,49149
LD A,63 ; Set AY register 7 to 63 - i.e. all channels OFF
OUT (C),A
POP HL
POP DE
POP BC
POP AF
RETI’m not going through the sub routines of the loop, other than to note the following:
- UPDATE is a whole series of instructions that basically do the following to output the HEX value of a sound parameter:
LD A, (contents of one of the sound variables)
LD DE, (corresponding screen address for the variable to be displayed)
CALL HEX- REFRESH runs the sound driver itself, as described in Z80 and AY-3-8910.
- CLOCK decrements the FIFTY variable and every time it gets to zero updates SECS and MINS and writes them out to the display. As it also uses the HEX routine, I guess it is storing the time using binary-coded decimal (BCD).
- KEYSCAN reads the last key pressed from the system variable location stored in ASCII (23560 / 0x5C08).
At some point I might come back and work out what keys do what…
Closing Thoughts
I’d really like to get some of this code running on some of the alternate Z80 platforms I have. Getting the sound output shouldn’t be too much of an issue, but I’d really like to have some kind of display too.
But as can be seen above, the tester UI is pretty well tied into the oddities of the ZX Spectrum display, so porting it won’t be trivial.
I suspect there are already some existing AY/chiptune players that perhaps would be a better starting point, but from what I’ve seen they tend to stream the register data after having sampled it at regular intervals, which isn’t quite what I was after… there would be something really quite interesting about actually running Ste Ruddy’s Sound Driver with a Tim Follin soundtrack programmed in.
Kevin
#ay38910 #TimFollin #zxSpectrum -
Z80 and AY-3-8910 – Part 2
I’ve spent a bit of time looking at the “Tester” part of the AY driver code for Tim Follin’s music archive that I talked about in Z80 and AY-3-8910.
This is documenting what I think I’ve worked out so far for the tester code.
The Sound Tester
As previously mentioned, there are essentially three parts to the code in Follin archive:
- The tune and effect data.
- Ste Ruddy’s Sound Driver.
- A tracker-style (ish) tester UI application.
The first part looked at the sound driver itself, and essentially skipped over the tester part of the code. This post picks up on that tester code.
Reminder, from part one, the main structure is as follows:
Code_Start: EQU 40000
Data_Start: EQU 50000
;-----------------------------
ORG Code_Start
; The UI/tester code
TESTER:
LOOP: Calls the following for each scan:
HALT - Suspends until an interrupt comes in?
CALL UPDATE
CALL REFRESH
CALL CLOCK
CALL KEYSCAN
Repeat as necessary
KEYSCAN: UI scanning
CLOCK: Possibly maintain a 50Hz refresh rate clock?
UPDATE: Loads the internal state of all sound variables from
the driver and displays them in real time via the UI.
; The sound driver
CODE_TOP:
TUNE: Select which tune to play.
TUNE_IN: Init all internal sound state variables for a new tune.
TUNEOFF: Stop a playing tune, eg to change tune or start an FX.
FX: Start playing an FX.
FLOOP: Keep processing FX instructions until complete.
REFRESH: "run" a scan of the sound driver updating and outputting the soundThe Tester Code
Initialisation information and main screen data:
;**************************************
; Z80 AY MUSIC DRIVER
;**************************************
; ORG 40000
; LOAD 0C000H
;======================================
;STACK DEPTHS
SD: EQU 3
;======================================
ASCII: EQU 23560 ; 23560 = $5C08 = System Variable "LAST K"
TESTER: PUSH AF
PUSH BC
PUSH DE
PUSH HL
XOR A ; ASCII = MINS = SECS = 0
LD (ASCII),A
LD (MINS),A
LD (SECS),A
CALL TUNEOFF ; TUNE initialisation
CALL STACKMESS ; Kick off the Tester code!
DB CLS ; The start of the main UI data
DB AT,0,0
DB INK,01010111B
DB "'AY' MUSIC DRIVE"
DB "R V2 BY S.RUDDY"
... Skip ...
DB INK,64+5
DB "VOLUME "
DB " "
DB 255
... Skip ...
AT: EQU 22
INK: EQU 16
CLS: EQU 15
STACKMESS: POP IX
CALL MESS
JP (IX)There is a whole lot of screen data in DB blocks which includes some “op codes” that are defined later: AT, INK, CLS. These are special codes that are used by the ROM-based print routines (more here), as used by Sinclair BASIC, but in this case they are spelt out directly, later in code. The final 255 signifies the end of the screen data.
So how are these definitions handled? That all comes up in the “MESS” routine I’ll get to in a moment, but first that “STACKMESS” routine needs a bit of explanation.
When a CALL instruction happens, such as the CALL STACKMESS at the start, the current program counter gets pushed onto the stack. In this case the current PC will point to the instruction after the CALL, which happens to be the start of the screen data. So the POP IX will grab the address of the screen data and drop it into IX and then call the “MESS” function to actually get on with it!
But before I get to that, there is some more code after the screen data:
LD HL,CALC1
PUSH HL
LD A,H
LD DE,4067H ; Output high byte
CALL HEX
POP HL
LD A,L
LD DE,4069H ; Output low byte
CALL HEX
LD HL,(CALC2)
PUSH HL
LD A,H
LD DE,4071H ; Output number of Tunes
CALL HEX
POP HL
LD A,L
LD DE,4073H ; Output number of effects
CALL HEX
LD HL,CALC1
LD DE,(CALC2)
ADD HL,DE
PUSH HL
LD A,H
LD DE,407CH ; Not entirely sure what this is outputting...
CALL HEX
POP HL
LD A,L
LD DE,407EH
CALL HEXThis is writing some basic data out to the display. CALC1 seems to relate to code section size. I believe CALC2 is the start address of the tune data, which is the following:
ORG Data_Start
TUNES: EQU 5
EFFECTS: EQU 21All three of these sections are outputting a 16-bit value in two single-byte chunks using the “HEX” routine, which takes a screen address (in the range $4000-$57FF) and outputs a hex number at that screen location.
So while I’m at it then, how is that HEX function working?
;--------------------------------------
HEX: INC DE ; DE contains the screen address to use
PUSH AF ; Start with DE+1
CALL ONEnib ; Write out the LOW 4-bits
POP AF
RRA ; A = A>>4
RRA ; to write out HIGH 4-bits
RRA
RRA
DEC DE ; Back to original DE screen address
ONEnib: AND 15 ; A = A & 0xF
ADD A ; BC = A * 2
LD C,A
LD B,0
LD HL,ROM_TAB ; Read from ROM_TAB[BC]
ADD HL,BC
LD A,(HL)
INC HL
LD H,(HL)
LD L,A ; HL = (uint16_t)ROM_TAB[A]
MIKESbug: LD C,D ; So HL now points to character bitmap in ROM
LD B,8 ; Write out 8 bytes to display memory directly
PRloop: LD A,(HL) ; (DE) = (HL)
LD (DE),A
INC HL ; HL++
INC D ; NB: Layout of display mem means D++ is next line of char
; for same value of E.
DJNZ PRloop ; WHILE (B-- > 0)
LD D,C ; (Restore D before returning, so DE still = screen addr)
RET
ROM_TAB: DW 3D80H ; ROM character set: 3D80 = "0"
DW 3D88H ; Each char = 8 x 8 bits
DW 3D90H
DW 3D98H
DW 3DA0H
DW 3DA8H
DW 3DB0H
DW 3DB8H
DW 3DC0H
DW 3DC8H ; = "9"
DW 3E08H ; = "A"
DW 3E10H
DW 3E18H
DW 3E20H
DW 3E28H
DW 3E30H ; = "F"This is making use of the character set stored in the Spectrum ROM (more here) which is indexed via a 16-word jump table mapping the characters onto each of the 16 hex characters: 0..9, A..F.
Then each byte, 8 in total, of the character is written directly out to the Spectrum screen memory taking advantage of the odd formatting of the screen memory to easily skip to the next line of the display for each line of the character (more here).
So before I get into the main update loop, how the screen initialised and set up? That happens in the “MESS” and some ancillary functions.
MESS: LD A,(IX+0) ; At this point, McursorX, McursorY = (0,0)
INC IX ; So read a byte of screen data
OR A
RET M ; Stop IF A=255 (i.e. negative)
CP 32
JR C,Mcontrol ; IF A<32 process control character then RET back to "MESS"
CALL Mgetchar ; ELSE Process character
CALL Mgetaddr ; Get screen address for next output in DE
CALL MIKESbug ; Output the character
CALL PRattr ; Set the colour attributes
CALL INCcursor ; Update the screen position for the next byte of screen data
JR MESS
Mcontrol: LD HL,MESS ; Stick the address of "MESS" on the stack for the RET
PUSH HL
CP 15 ; IF A == CLS
JR Z,Mcls
CP 22 ; IF A == AT
JP Z,Mat
CP 16 ; IF A == INK
JR Z,Mink
RET ; RETurn to "MESS"
Mcolour: DB 0 ; Working variables for cursor position and colour
McursorX: DB 0
McursorY: DB 0 ; Has to be directly after McursorX (see later)
Mink: LD A,(IX+0) ; Process INK to set colour
INC IX
LD (Mcolour),A
RET
Mcls: LD HL,4000H ; Process CLS to clear screen
LD (HL),L
LD DE,4001H
LD BC,1AFFH
LDIR
LD (McursorX),BC
RET
INCcursor: LD HL,McursorX ; Moves the cursor on one position
LD A,(HL)
INC A
AND 31
LD (HL),A ; X++; X = X % 32
RET NZ ; IF X==0; Y++
INC HL ; Assumes McursorY is McursorX++
INC (HL)
RET
Mgetchar: LD L,A ; HL = A*8 + 3C00
LD H,0 ; Note: A > 32; where 32="Space"
ADD HL,HL ; In ROM, space is address 3D00
ADD HL,HL ; 32 * 8 = 0x100
ADD HL,HL
LD BC,3C00H
ADD HL,BC ; HL = Start address of character map for char in A in ROM
RET
.... skip ....
Mgetaddr: LD A,(McursorY) ; Calculate the screen address for (McursorX, McursorY)
AND 18H
OR 40H
LD D,A
LD A,(McursorY)
RRCA
RRCA
RRCA
AND 0E0H
LD E,A
LD A,(McursorX)
ADD E
LD E,A
RET ; DE = required screen address
Mat: LD A,(IX+0) ; Set cursor to provided X, Y in screen data
LD (McursorX),A
INC IX
LD A,(IX+0)
LD (McursorY),A
INC IX
RET
PRattr: LD A,D ; Get address of ATTRibute memory
RRA
RRA
RRA
AND 3
OR 58H
LD D,A
LD A,(Mcolour)
LD (DE),A ; And set the colour
RETBasically this loop keeps working on the provided screen data until the value 255 is found, at which point it returns. There are two paths for handling the data:
- IF the value is < 32 then it is a control value. Only CLS, AT and INK are recognised.
- ELSE the value is assumed to be an ASCII character and is displayed.
Whatever is happening, happens at the coordinates given by (McursorX, McursorY) which start out as (0,0) and get updated automatically when a character is output, or in response to an AT command. INK will set the required colour in Mcolour, which again starts out as 0. This is applied after the character is written to the screen, using the PRattr function.
There is a fun bit of optimisation going on in Mcontrol. At the start it pushes the address of the MESS function on the stack, which means that the RET will jump back to the start of MESS rather than where the jump happened to Mcontrol itself.
There is another shortcut in the Mcls function: LDIR. From http://z80-heaven.wikidot.com/instructions-set:ldir: “Repeats LDI (LD (DE),(HL), then increments DE, HL, and decrements BC) until BC=0.” By setting the contents of HL (the first byte of the display) to zero, this will tile that same value across the display memory until BC, which starts at $1AFF, is zero. This will zero the whole display – both pixels and attributes – from 0x4000 through to 0x5AFF.
Now finally, we get to the main update loop.
LOOP:
HALT
CALL UPDATE ; Update the display from the current Sound parameters
LD A,2
OUT (254),A ; Set border to 2
CALL REFRESH ; Update the sound driver parameters
XOR A
OUT (254),A ; Set border to 0
CALL CLOCK ; Run 50Hz clock
CALL KEYSCAN ; Guess what - scans the keyboard 🙂
LD A,07FH
IN A,(254) ; Reads 0x7FFE which is the bottom row of the keyboard
AND 1
JP NZ,LOOP ; Checks bit 0, which is the SPACE key
LD BC,65533 ; AY OUTPUT PORTS (FFFD, BFFD)
LD A,7
OUT (C),A
LD BC,49149
LD A,63 ; Set AY register 7 to 63 - i.e. all channels OFF
OUT (C),A
POP HL
POP DE
POP BC
POP AF
RETI’m not going through the sub routines of the loop, other than to note the following:
- UPDATE is a whole series of instructions that basically do the following to output the HEX value of a sound parameter:
LD A, (contents of one of the sound variables)
LD DE, (corresponding screen address for the variable to be displayed)
CALL HEX- REFRESH runs the sound driver itself, as described in Z80 and AY-3-8910.
- CLOCK decrements the FIFTY variable and every time it gets to zero updates SECS and MINS and writes them out to the display. As it also uses the HEX routine, I guess it is storing the time using binary-coded decimal (BCD).
- KEYSCAN reads the last key pressed from the system variable location stored in ASCII (23560 / 0x5C08).
At some point I might come back and work out what keys do what…
Closing Thoughts
I’d really like to get some of this code running on some of the alternate Z80 platforms I have. Getting the sound output shouldn’t be too much of an issue, but I’d really like to have some kind of display too.
But as can be seen above, the tester UI is pretty well tied into the oddities of the ZX Spectrum display, so porting it won’t be trivial.
I suspect there are already some existing AY/chiptune players that perhaps would be a better starting point, but from what I’ve seen they tend to stream the register data after having sampled it at regular intervals, which isn’t quite what I was after… there would be something really quite interesting about actually running Ste Ruddy’s Sound Driver with a Tim Follin soundtrack programmed in.
Kevin
#ay38910 #TimFollin #zxSpectrum -
Z80 and AY-3-8910 – Part 2
I’ve spent a bit of time looking at the “Tester” part of the AY driver code for Tim Follin’s music archive that I talked about in Z80 and AY-3-8910.
This is documenting what I think I’ve worked out so far for the tester code.
The Sound Tester
As previously mentioned, there are essentially three parts to the code in Follin archive:
- The tune and effect data.
- Ste Ruddy’s Sound Driver.
- A tracker-style (ish) tester UI application.
The first part looked at the sound driver itself, and essentially skipped over the tester part of the code. This post picks up on that tester code.
Reminder, from part one, the main structure is as follows:
Code_Start: EQU 40000
Data_Start: EQU 50000
;-----------------------------
ORG Code_Start
; The UI/tester code
TESTER:
LOOP: Calls the following for each scan:
HALT - Suspends until an interrupt comes in?
CALL UPDATE
CALL REFRESH
CALL CLOCK
CALL KEYSCAN
Repeat as necessary
KEYSCAN: UI scanning
CLOCK: Possibly maintain a 50Hz refresh rate clock?
UPDATE: Loads the internal state of all sound variables from
the driver and displays them in real time via the UI.
; The sound driver
CODE_TOP:
TUNE: Select which tune to play.
TUNE_IN: Init all internal sound state variables for a new tune.
TUNEOFF: Stop a playing tune, eg to change tune or start an FX.
FX: Start playing an FX.
FLOOP: Keep processing FX instructions until complete.
REFRESH: "run" a scan of the sound driver updating and outputting the soundThe Tester Code
Initialisation information and main screen data:
;**************************************
; Z80 AY MUSIC DRIVER
;**************************************
; ORG 40000
; LOAD 0C000H
;======================================
;STACK DEPTHS
SD: EQU 3
;======================================
ASCII: EQU 23560 ; 23560 = $5C08 = System Variable "LAST K"
TESTER: PUSH AF
PUSH BC
PUSH DE
PUSH HL
XOR A ; ASCII = MINS = SECS = 0
LD (ASCII),A
LD (MINS),A
LD (SECS),A
CALL TUNEOFF ; TUNE initialisation
CALL STACKMESS ; Kick off the Tester code!
DB CLS ; The start of the main UI data
DB AT,0,0
DB INK,01010111B
DB "'AY' MUSIC DRIVE"
DB "R V2 BY S.RUDDY"
... Skip ...
DB INK,64+5
DB "VOLUME "
DB " "
DB 255
... Skip ...
AT: EQU 22
INK: EQU 16
CLS: EQU 15
STACKMESS: POP IX
CALL MESS
JP (IX)There is a whole lot of screen data in DB blocks which includes some “op codes” that are defined later: AT, INK, CLS. These are special codes that are used by the ROM-based print routines (more here), as used by Sinclair BASIC, but in this case they are spelt out directly, later in code. The final 255 signifies the end of the screen data.
So how are these definitions handled? That all comes up in the “MESS” routine I’ll get to in a moment, but first that “STACKMESS” routine needs a bit of explanation.
When a CALL instruction happens, such as the CALL STACKMESS at the start, the current program counter gets pushed onto the stack. In this case the current PC will point to the instruction after the CALL, which happens to be the start of the screen data. So the POP IX will grab the address of the screen data and drop it into IX and then call the “MESS” function to actually get on with it!
But before I get to that, there is some more code after the screen data:
LD HL,CALC1
PUSH HL
LD A,H
LD DE,4067H ; Output high byte
CALL HEX
POP HL
LD A,L
LD DE,4069H ; Output low byte
CALL HEX
LD HL,(CALC2)
PUSH HL
LD A,H
LD DE,4071H ; Output number of Tunes
CALL HEX
POP HL
LD A,L
LD DE,4073H ; Output number of effects
CALL HEX
LD HL,CALC1
LD DE,(CALC2)
ADD HL,DE
PUSH HL
LD A,H
LD DE,407CH ; Not entirely sure what this is outputting...
CALL HEX
POP HL
LD A,L
LD DE,407EH
CALL HEXThis is writing some basic data out to the display. CALC1 seems to relate to code section size. I believe CALC2 is the start address of the tune data, which is the following:
ORG Data_Start
TUNES: EQU 5
EFFECTS: EQU 21All three of these sections are outputting a 16-bit value in two single-byte chunks using the “HEX” routine, which takes a screen address (in the range $4000-$57FF) and outputs a hex number at that screen location.
So while I’m at it then, how is that HEX function working?
;--------------------------------------
HEX: INC DE ; DE contains the screen address to use
PUSH AF ; Start with DE+1
CALL ONEnib ; Write out the LOW 4-bits
POP AF
RRA ; A = A>>4
RRA ; to write out HIGH 4-bits
RRA
RRA
DEC DE ; Back to original DE screen address
ONEnib: AND 15 ; A = A & 0xF
ADD A ; BC = A * 2
LD C,A
LD B,0
LD HL,ROM_TAB ; Read from ROM_TAB[BC]
ADD HL,BC
LD A,(HL)
INC HL
LD H,(HL)
LD L,A ; HL = (uint16_t)ROM_TAB[A]
MIKESbug: LD C,D ; So HL now points to character bitmap in ROM
LD B,8 ; Write out 8 bytes to display memory directly
PRloop: LD A,(HL) ; (DE) = (HL)
LD (DE),A
INC HL ; HL++
INC D ; NB: Layout of display mem means D++ is next line of char
; for same value of E.
DJNZ PRloop ; WHILE (B-- > 0)
LD D,C ; (Restore D before returning, so DE still = screen addr)
RET
ROM_TAB: DW 3D80H ; ROM character set: 3D80 = "0"
DW 3D88H ; Each char = 8 x 8 bits
DW 3D90H
DW 3D98H
DW 3DA0H
DW 3DA8H
DW 3DB0H
DW 3DB8H
DW 3DC0H
DW 3DC8H ; = "9"
DW 3E08H ; = "A"
DW 3E10H
DW 3E18H
DW 3E20H
DW 3E28H
DW 3E30H ; = "F"This is making use of the character set stored in the Spectrum ROM (more here) which is indexed via a 16-word jump table mapping the characters onto each of the 16 hex characters: 0..9, A..F.
Then each byte, 8 in total, of the character is written directly out to the Spectrum screen memory taking advantage of the odd formatting of the screen memory to easily skip to the next line of the display for each line of the character (more here).
So before I get into the main update loop, how the screen initialised and set up? That happens in the “MESS” and some ancillary functions.
MESS: LD A,(IX+0) ; At this point, McursorX, McursorY = (0,0)
INC IX ; So read a byte of screen data
OR A
RET M ; Stop IF A=255 (i.e. negative)
CP 32
JR C,Mcontrol ; IF A<32 process control character then RET back to "MESS"
CALL Mgetchar ; ELSE Process character
CALL Mgetaddr ; Get screen address for next output in DE
CALL MIKESbug ; Output the character
CALL PRattr ; Set the colour attributes
CALL INCcursor ; Update the screen position for the next byte of screen data
JR MESS
Mcontrol: LD HL,MESS ; Stick the address of "MESS" on the stack for the RET
PUSH HL
CP 15 ; IF A == CLS
JR Z,Mcls
CP 22 ; IF A == AT
JP Z,Mat
CP 16 ; IF A == INK
JR Z,Mink
RET ; RETurn to "MESS"
Mcolour: DB 0 ; Working variables for cursor position and colour
McursorX: DB 0
McursorY: DB 0 ; Has to be directly after McursorX (see later)
Mink: LD A,(IX+0) ; Process INK to set colour
INC IX
LD (Mcolour),A
RET
Mcls: LD HL,4000H ; Process CLS to clear screen
LD (HL),L
LD DE,4001H
LD BC,1AFFH
LDIR
LD (McursorX),BC
RET
INCcursor: LD HL,McursorX ; Moves the cursor on one position
LD A,(HL)
INC A
AND 31
LD (HL),A ; X++; X = X % 32
RET NZ ; IF X==0; Y++
INC HL ; Assumes McursorY is McursorX++
INC (HL)
RET
Mgetchar: LD L,A ; HL = A*8 + 3C00
LD H,0 ; Note: A > 32; where 32="Space"
ADD HL,HL ; In ROM, space is address 3D00
ADD HL,HL ; 32 * 8 = 0x100
ADD HL,HL
LD BC,3C00H
ADD HL,BC ; HL = Start address of character map for char in A in ROM
RET
.... skip ....
Mgetaddr: LD A,(McursorY) ; Calculate the screen address for (McursorX, McursorY)
AND 18H
OR 40H
LD D,A
LD A,(McursorY)
RRCA
RRCA
RRCA
AND 0E0H
LD E,A
LD A,(McursorX)
ADD E
LD E,A
RET ; DE = required screen address
Mat: LD A,(IX+0) ; Set cursor to provided X, Y in screen data
LD (McursorX),A
INC IX
LD A,(IX+0)
LD (McursorY),A
INC IX
RET
PRattr: LD A,D ; Get address of ATTRibute memory
RRA
RRA
RRA
AND 3
OR 58H
LD D,A
LD A,(Mcolour)
LD (DE),A ; And set the colour
RETBasically this loop keeps working on the provided screen data until the value 255 is found, at which point it returns. There are two paths for handling the data:
- IF the value is < 32 then it is a control value. Only CLS, AT and INK are recognised.
- ELSE the value is assumed to be an ASCII character and is displayed.
Whatever is happening, happens at the coordinates given by (McursorX, McursorY) which start out as (0,0) and get updated automatically when a character is output, or in response to an AT command. INK will set the required colour in Mcolour, which again starts out as 0. This is applied after the character is written to the screen, using the PRattr function.
There is a fun bit of optimisation going on in Mcontrol. At the start it pushes the address of the MESS function on the stack, which means that the RET will jump back to the start of MESS rather than where the jump happened to Mcontrol itself.
There is another shortcut in the Mcls function: LDIR. From http://z80-heaven.wikidot.com/instructions-set:ldir: “Repeats LDI (LD (DE),(HL), then increments DE, HL, and decrements BC) until BC=0.” By setting the contents of HL (the first byte of the display) to zero, this will tile that same value across the display memory until BC, which starts at $1AFF, is zero. This will zero the whole display – both pixels and attributes – from 0x4000 through to 0x5AFF.
Now finally, we get to the main update loop.
LOOP:
HALT
CALL UPDATE ; Update the display from the current Sound parameters
LD A,2
OUT (254),A ; Set border to 2
CALL REFRESH ; Update the sound driver parameters
XOR A
OUT (254),A ; Set border to 0
CALL CLOCK ; Run 50Hz clock
CALL KEYSCAN ; Guess what - scans the keyboard 🙂
LD A,07FH
IN A,(254) ; Reads 0x7FFE which is the bottom row of the keyboard
AND 1
JP NZ,LOOP ; Checks bit 0, which is the SPACE key
LD BC,65533 ; AY OUTPUT PORTS (FFFD, BFFD)
LD A,7
OUT (C),A
LD BC,49149
LD A,63 ; Set AY register 7 to 63 - i.e. all channels OFF
OUT (C),A
POP HL
POP DE
POP BC
POP AF
RETI’m not going through the sub routines of the loop, other than to note the following:
- UPDATE is a whole series of instructions that basically do the following to output the HEX value of a sound parameter:
LD A, (contents of one of the sound variables)
LD DE, (corresponding screen address for the variable to be displayed)
CALL HEX- REFRESH runs the sound driver itself, as described in Z80 and AY-3-8910.
- CLOCK decrements the FIFTY variable and every time it gets to zero updates SECS and MINS and writes them out to the display. As it also uses the HEX routine, I guess it is storing the time using binary-coded decimal (BCD).
- KEYSCAN reads the last key pressed from the system variable location stored in ASCII (23560 / 0x5C08).
At some point I might come back and work out what keys do what…
Closing Thoughts
I’d really like to get some of this code running on some of the alternate Z80 platforms I have. Getting the sound output shouldn’t be too much of an issue, but I’d really like to have some kind of display too.
But as can be seen above, the tester UI is pretty well tied into the oddities of the ZX Spectrum display, so porting it won’t be trivial.
I suspect there are already some existing AY/chiptune players that perhaps would be a better starting point, but from what I’ve seen they tend to stream the register data after having sampled it at regular intervals, which isn’t quite what I was after… there would be something really quite interesting about actually running Ste Ruddy’s Sound Driver with a Tim Follin soundtrack programmed in.
Kevin
#ay38910 #TimFollin #zxSpectrum -
Z80 and AY-3-8910 – Part 2
I’ve spent a bit of time looking at the “Tester” part of the AY driver code for Tim Follin’s music archive that I talked about in Z80 and AY-3-8910.
This is documenting what I think I’ve worked out so far for the tester code.
The Sound Tester
As previously mentioned, there are essentially three parts to the code in Follin archive:
- The tune and effect data.
- Ste Ruddy’s Sound Driver.
- A tracker-style (ish) tester UI application.
The first part looked at the sound driver itself, and essentially skipped over the tester part of the code. This post picks up on that tester code.
Reminder, from part one, the main structure is as follows:
Code_Start: EQU 40000
Data_Start: EQU 50000
;-----------------------------
ORG Code_Start
; The UI/tester code
TESTER:
LOOP: Calls the following for each scan:
HALT - Suspends until an interrupt comes in?
CALL UPDATE
CALL REFRESH
CALL CLOCK
CALL KEYSCAN
Repeat as necessary
KEYSCAN: UI scanning
CLOCK: Possibly maintain a 50Hz refresh rate clock?
UPDATE: Loads the internal state of all sound variables from
the driver and displays them in real time via the UI.
; The sound driver
CODE_TOP:
TUNE: Select which tune to play.
TUNE_IN: Init all internal sound state variables for a new tune.
TUNEOFF: Stop a playing tune, eg to change tune or start an FX.
FX: Start playing an FX.
FLOOP: Keep processing FX instructions until complete.
REFRESH: "run" a scan of the sound driver updating and outputting the soundThe Tester Code
Initialisation information and main screen data:
;**************************************
; Z80 AY MUSIC DRIVER
;**************************************
; ORG 40000
; LOAD 0C000H
;======================================
;STACK DEPTHS
SD: EQU 3
;======================================
ASCII: EQU 23560 ; 23560 = $5C08 = System Variable "LAST K"
TESTER: PUSH AF
PUSH BC
PUSH DE
PUSH HL
XOR A ; ASCII = MINS = SECS = 0
LD (ASCII),A
LD (MINS),A
LD (SECS),A
CALL TUNEOFF ; TUNE initialisation
CALL STACKMESS ; Kick off the Tester code!
DB CLS ; The start of the main UI data
DB AT,0,0
DB INK,01010111B
DB "'AY' MUSIC DRIVE"
DB "R V2 BY S.RUDDY"
... Skip ...
DB INK,64+5
DB "VOLUME "
DB " "
DB 255
... Skip ...
AT: EQU 22
INK: EQU 16
CLS: EQU 15
STACKMESS: POP IX
CALL MESS
JP (IX)There is a whole lot of screen data in DB blocks which includes some “op codes” that are defined later: AT, INK, CLS. These are special codes that are used by the ROM-based print routines (more here), as used by Sinclair BASIC, but in this case they are spelt out directly, later in code. The final 255 signifies the end of the screen data.
So how are these definitions handled? That all comes up in the “MESS” routine I’ll get to in a moment, but first that “STACKMESS” routine needs a bit of explanation.
When a CALL instruction happens, such as the CALL STACKMESS at the start, the current program counter gets pushed onto the stack. In this case the current PC will point to the instruction after the CALL, which happens to be the start of the screen data. So the POP IX will grab the address of the screen data and drop it into IX and then call the “MESS” function to actually get on with it!
But before I get to that, there is some more code after the screen data:
LD HL,CALC1
PUSH HL
LD A,H
LD DE,4067H ; Output high byte
CALL HEX
POP HL
LD A,L
LD DE,4069H ; Output low byte
CALL HEX
LD HL,(CALC2)
PUSH HL
LD A,H
LD DE,4071H ; Output number of Tunes
CALL HEX
POP HL
LD A,L
LD DE,4073H ; Output number of effects
CALL HEX
LD HL,CALC1
LD DE,(CALC2)
ADD HL,DE
PUSH HL
LD A,H
LD DE,407CH ; Not entirely sure what this is outputting...
CALL HEX
POP HL
LD A,L
LD DE,407EH
CALL HEXThis is writing some basic data out to the display. CALC1 seems to relate to code section size. I believe CALC2 is the start address of the tune data, which is the following:
ORG Data_Start
TUNES: EQU 5
EFFECTS: EQU 21All three of these sections are outputting a 16-bit value in two single-byte chunks using the “HEX” routine, which takes a screen address (in the range $4000-$57FF) and outputs a hex number at that screen location.
So while I’m at it then, how is that HEX function working?
;--------------------------------------
HEX: INC DE ; DE contains the screen address to use
PUSH AF ; Start with DE+1
CALL ONEnib ; Write out the LOW 4-bits
POP AF
RRA ; A = A>>4
RRA ; to write out HIGH 4-bits
RRA
RRA
DEC DE ; Back to original DE screen address
ONEnib: AND 15 ; A = A & 0xF
ADD A ; BC = A * 2
LD C,A
LD B,0
LD HL,ROM_TAB ; Read from ROM_TAB[BC]
ADD HL,BC
LD A,(HL)
INC HL
LD H,(HL)
LD L,A ; HL = (uint16_t)ROM_TAB[A]
MIKESbug: LD C,D ; So HL now points to character bitmap in ROM
LD B,8 ; Write out 8 bytes to display memory directly
PRloop: LD A,(HL) ; (DE) = (HL)
LD (DE),A
INC HL ; HL++
INC D ; NB: Layout of display mem means D++ is next line of char
; for same value of E.
DJNZ PRloop ; WHILE (B-- > 0)
LD D,C ; (Restore D before returning, so DE still = screen addr)
RET
ROM_TAB: DW 3D80H ; ROM character set: 3D80 = "0"
DW 3D88H ; Each char = 8 x 8 bits
DW 3D90H
DW 3D98H
DW 3DA0H
DW 3DA8H
DW 3DB0H
DW 3DB8H
DW 3DC0H
DW 3DC8H ; = "9"
DW 3E08H ; = "A"
DW 3E10H
DW 3E18H
DW 3E20H
DW 3E28H
DW 3E30H ; = "F"This is making use of the character set stored in the Spectrum ROM (more here) which is indexed via a 16-word jump table mapping the characters onto each of the 16 hex characters: 0..9, A..F.
Then each byte, 8 in total, of the character is written directly out to the Spectrum screen memory taking advantage of the odd formatting of the screen memory to easily skip to the next line of the display for each line of the character (more here).
So before I get into the main update loop, how the screen initialised and set up? That happens in the “MESS” and some ancillary functions.
MESS: LD A,(IX+0) ; At this point, McursorX, McursorY = (0,0)
INC IX ; So read a byte of screen data
OR A
RET M ; Stop IF A=255 (i.e. negative)
CP 32
JR C,Mcontrol ; IF A<32 process control character then RET back to "MESS"
CALL Mgetchar ; ELSE Process character
CALL Mgetaddr ; Get screen address for next output in DE
CALL MIKESbug ; Output the character
CALL PRattr ; Set the colour attributes
CALL INCcursor ; Update the screen position for the next byte of screen data
JR MESS
Mcontrol: LD HL,MESS ; Stick the address of "MESS" on the stack for the RET
PUSH HL
CP 15 ; IF A == CLS
JR Z,Mcls
CP 22 ; IF A == AT
JP Z,Mat
CP 16 ; IF A == INK
JR Z,Mink
RET ; RETurn to "MESS"
Mcolour: DB 0 ; Working variables for cursor position and colour
McursorX: DB 0
McursorY: DB 0 ; Has to be directly after McursorX (see later)
Mink: LD A,(IX+0) ; Process INK to set colour
INC IX
LD (Mcolour),A
RET
Mcls: LD HL,4000H ; Process CLS to clear screen
LD (HL),L
LD DE,4001H
LD BC,1AFFH
LDIR
LD (McursorX),BC
RET
INCcursor: LD HL,McursorX ; Moves the cursor on one position
LD A,(HL)
INC A
AND 31
LD (HL),A ; X++; X = X % 32
RET NZ ; IF X==0; Y++
INC HL ; Assumes McursorY is McursorX++
INC (HL)
RET
Mgetchar: LD L,A ; HL = A*8 + 3C00
LD H,0 ; Note: A > 32; where 32="Space"
ADD HL,HL ; In ROM, space is address 3D00
ADD HL,HL ; 32 * 8 = 0x100
ADD HL,HL
LD BC,3C00H
ADD HL,BC ; HL = Start address of character map for char in A in ROM
RET
.... skip ....
Mgetaddr: LD A,(McursorY) ; Calculate the screen address for (McursorX, McursorY)
AND 18H
OR 40H
LD D,A
LD A,(McursorY)
RRCA
RRCA
RRCA
AND 0E0H
LD E,A
LD A,(McursorX)
ADD E
LD E,A
RET ; DE = required screen address
Mat: LD A,(IX+0) ; Set cursor to provided X, Y in screen data
LD (McursorX),A
INC IX
LD A,(IX+0)
LD (McursorY),A
INC IX
RET
PRattr: LD A,D ; Get address of ATTRibute memory
RRA
RRA
RRA
AND 3
OR 58H
LD D,A
LD A,(Mcolour)
LD (DE),A ; And set the colour
RETBasically this loop keeps working on the provided screen data until the value 255 is found, at which point it returns. There are two paths for handling the data:
- IF the value is < 32 then it is a control value. Only CLS, AT and INK are recognised.
- ELSE the value is assumed to be an ASCII character and is displayed.
Whatever is happening, happens at the coordinates given by (McursorX, McursorY) which start out as (0,0) and get updated automatically when a character is output, or in response to an AT command. INK will set the required colour in Mcolour, which again starts out as 0. This is applied after the character is written to the screen, using the PRattr function.
There is a fun bit of optimisation going on in Mcontrol. At the start it pushes the address of the MESS function on the stack, which means that the RET will jump back to the start of MESS rather than where the jump happened to Mcontrol itself.
There is another shortcut in the Mcls function: LDIR. From http://z80-heaven.wikidot.com/instructions-set:ldir: “Repeats LDI (LD (DE),(HL), then increments DE, HL, and decrements BC) until BC=0.” By setting the contents of HL (the first byte of the display) to zero, this will tile that same value across the display memory until BC, which starts at $1AFF, is zero. This will zero the whole display – both pixels and attributes – from 0x4000 through to 0x5AFF.
Now finally, we get to the main update loop.
LOOP:
HALT
CALL UPDATE ; Update the display from the current Sound parameters
LD A,2
OUT (254),A ; Set border to 2
CALL REFRESH ; Update the sound driver parameters
XOR A
OUT (254),A ; Set border to 0
CALL CLOCK ; Run 50Hz clock
CALL KEYSCAN ; Guess what - scans the keyboard 🙂
LD A,07FH
IN A,(254) ; Reads 0x7FFE which is the bottom row of the keyboard
AND 1
JP NZ,LOOP ; Checks bit 0, which is the SPACE key
LD BC,65533 ; AY OUTPUT PORTS (FFFD, BFFD)
LD A,7
OUT (C),A
LD BC,49149
LD A,63 ; Set AY register 7 to 63 - i.e. all channels OFF
OUT (C),A
POP HL
POP DE
POP BC
POP AF
RETI’m not going through the sub routines of the loop, other than to note the following:
- UPDATE is a whole series of instructions that basically do the following to output the HEX value of a sound parameter:
LD A, (contents of one of the sound variables)
LD DE, (corresponding screen address for the variable to be displayed)
CALL HEX- REFRESH runs the sound driver itself, as described in Z80 and AY-3-8910.
- CLOCK decrements the FIFTY variable and every time it gets to zero updates SECS and MINS and writes them out to the display. As it also uses the HEX routine, I guess it is storing the time using binary-coded decimal (BCD).
- KEYSCAN reads the last key pressed from the system variable location stored in ASCII (23560 / 0x5C08).
At some point I might come back and work out what keys do what…
Closing Thoughts
I’d really like to get some of this code running on some of the alternate Z80 platforms I have. Getting the sound output shouldn’t be too much of an issue, but I’d really like to have some kind of display too.
But as can be seen above, the tester UI is pretty well tied into the oddities of the ZX Spectrum display, so porting it won’t be trivial.
I suspect there are already some existing AY/chiptune players that perhaps would be a better starting point, but from what I’ve seen they tend to stream the register data after having sampled it at regular intervals, which isn’t quite what I was after… there would be something really quite interesting about actually running Ste Ruddy’s Sound Driver with a Tim Follin soundtrack programmed in.
Kevin
#ay38910 #TimFollin #zxSpectrum -
Z80 and AY-3-8910 – Part 2
I’ve spent a bit of time looking at the “Tester” part of the AY driver code for Tim Follin’s music archive that I talked about in Z80 and AY-3-8910.
This is documenting what I think I’ve worked out so far for the tester code.
The Sound Tester
As previously mentioned, there are essentially three parts to the code in Follin archive:
- The tune and effect data.
- Ste Ruddy’s Sound Driver.
- A tracker-style (ish) tester UI application.
The first part looked at the sound driver itself, and essentially skipped over the tester part of the code. This post picks up on that tester code.
Reminder, from part one, the main structure is as follows:
Code_Start: EQU 40000
Data_Start: EQU 50000
;-----------------------------
ORG Code_Start
; The UI/tester code
TESTER:
LOOP: Calls the following for each scan:
HALT - Suspends until an interrupt comes in?
CALL UPDATE
CALL REFRESH
CALL CLOCK
CALL KEYSCAN
Repeat as necessary
KEYSCAN: UI scanning
CLOCK: Possibly maintain a 50Hz refresh rate clock?
UPDATE: Loads the internal state of all sound variables from
the driver and displays them in real time via the UI.
; The sound driver
CODE_TOP:
TUNE: Select which tune to play.
TUNE_IN: Init all internal sound state variables for a new tune.
TUNEOFF: Stop a playing tune, eg to change tune or start an FX.
FX: Start playing an FX.
FLOOP: Keep processing FX instructions until complete.
REFRESH: "run" a scan of the sound driver updating and outputting the soundThe Tester Code
Initialisation information and main screen data:
;**************************************
; Z80 AY MUSIC DRIVER
;**************************************
; ORG 40000
; LOAD 0C000H
;======================================
;STACK DEPTHS
SD: EQU 3
;======================================
ASCII: EQU 23560 ; 23560 = $5C08 = System Variable "LAST K"
TESTER: PUSH AF
PUSH BC
PUSH DE
PUSH HL
XOR A ; ASCII = MINS = SECS = 0
LD (ASCII),A
LD (MINS),A
LD (SECS),A
CALL TUNEOFF ; TUNE initialisation
CALL STACKMESS ; Kick off the Tester code!
DB CLS ; The start of the main UI data
DB AT,0,0
DB INK,01010111B
DB "'AY' MUSIC DRIVE"
DB "R V2 BY S.RUDDY"
... Skip ...
DB INK,64+5
DB "VOLUME "
DB " "
DB 255
... Skip ...
AT: EQU 22
INK: EQU 16
CLS: EQU 15
STACKMESS: POP IX
CALL MESS
JP (IX)There is a whole lot of screen data in DB blocks which includes some “op codes” that are defined later: AT, INK, CLS. These are special codes that are used by the ROM-based print routines (more here), as used by Sinclair BASIC, but in this case they are spelt out directly, later in code. The final 255 signifies the end of the screen data.
So how are these definitions handled? That all comes up in the “MESS” routine I’ll get to in a moment, but first that “STACKMESS” routine needs a bit of explanation.
When a CALL instruction happens, such as the CALL STACKMESS at the start, the current program counter gets pushed onto the stack. In this case the current PC will point to the instruction after the CALL, which happens to be the start of the screen data. So the POP IX will grab the address of the screen data and drop it into IX and then call the “MESS” function to actually get on with it!
But before I get to that, there is some more code after the screen data:
LD HL,CALC1
PUSH HL
LD A,H
LD DE,4067H ; Output high byte
CALL HEX
POP HL
LD A,L
LD DE,4069H ; Output low byte
CALL HEX
LD HL,(CALC2)
PUSH HL
LD A,H
LD DE,4071H ; Output number of Tunes
CALL HEX
POP HL
LD A,L
LD DE,4073H ; Output number of effects
CALL HEX
LD HL,CALC1
LD DE,(CALC2)
ADD HL,DE
PUSH HL
LD A,H
LD DE,407CH ; Not entirely sure what this is outputting...
CALL HEX
POP HL
LD A,L
LD DE,407EH
CALL HEXThis is writing some basic data out to the display. CALC1 seems to relate to code section size. I believe CALC2 is the start address of the tune data, which is the following:
ORG Data_Start
TUNES: EQU 5
EFFECTS: EQU 21All three of these sections are outputting a 16-bit value in two single-byte chunks using the “HEX” routine, which takes a screen address (in the range $4000-$57FF) and outputs a hex number at that screen location.
So while I’m at it then, how is that HEX function working?
;--------------------------------------
HEX: INC DE ; DE contains the screen address to use
PUSH AF ; Start with DE+1
CALL ONEnib ; Write out the LOW 4-bits
POP AF
RRA ; A = A>>4
RRA ; to write out HIGH 4-bits
RRA
RRA
DEC DE ; Back to original DE screen address
ONEnib: AND 15 ; A = A & 0xF
ADD A ; BC = A * 2
LD C,A
LD B,0
LD HL,ROM_TAB ; Read from ROM_TAB[BC]
ADD HL,BC
LD A,(HL)
INC HL
LD H,(HL)
LD L,A ; HL = (uint16_t)ROM_TAB[A]
MIKESbug: LD C,D ; So HL now points to character bitmap in ROM
LD B,8 ; Write out 8 bytes to display memory directly
PRloop: LD A,(HL) ; (DE) = (HL)
LD (DE),A
INC HL ; HL++
INC D ; NB: Layout of display mem means D++ is next line of char
; for same value of E.
DJNZ PRloop ; WHILE (B-- > 0)
LD D,C ; (Restore D before returning, so DE still = screen addr)
RET
ROM_TAB: DW 3D80H ; ROM character set: 3D80 = "0"
DW 3D88H ; Each char = 8 x 8 bits
DW 3D90H
DW 3D98H
DW 3DA0H
DW 3DA8H
DW 3DB0H
DW 3DB8H
DW 3DC0H
DW 3DC8H ; = "9"
DW 3E08H ; = "A"
DW 3E10H
DW 3E18H
DW 3E20H
DW 3E28H
DW 3E30H ; = "F"This is making use of the character set stored in the Spectrum ROM (more here) which is indexed via a 16-word jump table mapping the characters onto each of the 16 hex characters: 0..9, A..F.
Then each byte, 8 in total, of the character is written directly out to the Spectrum screen memory taking advantage of the odd formatting of the screen memory to easily skip to the next line of the display for each line of the character (more here).
So before I get into the main update loop, how the screen initialised and set up? That happens in the “MESS” and some ancillary functions.
MESS: LD A,(IX+0) ; At this point, McursorX, McursorY = (0,0)
INC IX ; So read a byte of screen data
OR A
RET M ; Stop IF A=255 (i.e. negative)
CP 32
JR C,Mcontrol ; IF A<32 process control character then RET back to "MESS"
CALL Mgetchar ; ELSE Process character
CALL Mgetaddr ; Get screen address for next output in DE
CALL MIKESbug ; Output the character
CALL PRattr ; Set the colour attributes
CALL INCcursor ; Update the screen position for the next byte of screen data
JR MESS
Mcontrol: LD HL,MESS ; Stick the address of "MESS" on the stack for the RET
PUSH HL
CP 15 ; IF A == CLS
JR Z,Mcls
CP 22 ; IF A == AT
JP Z,Mat
CP 16 ; IF A == INK
JR Z,Mink
RET ; RETurn to "MESS"
Mcolour: DB 0 ; Working variables for cursor position and colour
McursorX: DB 0
McursorY: DB 0 ; Has to be directly after McursorX (see later)
Mink: LD A,(IX+0) ; Process INK to set colour
INC IX
LD (Mcolour),A
RET
Mcls: LD HL,4000H ; Process CLS to clear screen
LD (HL),L
LD DE,4001H
LD BC,1AFFH
LDIR
LD (McursorX),BC
RET
INCcursor: LD HL,McursorX ; Moves the cursor on one position
LD A,(HL)
INC A
AND 31
LD (HL),A ; X++; X = X % 32
RET NZ ; IF X==0; Y++
INC HL ; Assumes McursorY is McursorX++
INC (HL)
RET
Mgetchar: LD L,A ; HL = A*8 + 3C00
LD H,0 ; Note: A > 32; where 32="Space"
ADD HL,HL ; In ROM, space is address 3D00
ADD HL,HL ; 32 * 8 = 0x100
ADD HL,HL
LD BC,3C00H
ADD HL,BC ; HL = Start address of character map for char in A in ROM
RET
.... skip ....
Mgetaddr: LD A,(McursorY) ; Calculate the screen address for (McursorX, McursorY)
AND 18H
OR 40H
LD D,A
LD A,(McursorY)
RRCA
RRCA
RRCA
AND 0E0H
LD E,A
LD A,(McursorX)
ADD E
LD E,A
RET ; DE = required screen address
Mat: LD A,(IX+0) ; Set cursor to provided X, Y in screen data
LD (McursorX),A
INC IX
LD A,(IX+0)
LD (McursorY),A
INC IX
RET
PRattr: LD A,D ; Get address of ATTRibute memory
RRA
RRA
RRA
AND 3
OR 58H
LD D,A
LD A,(Mcolour)
LD (DE),A ; And set the colour
RETBasically this loop keeps working on the provided screen data until the value 255 is found, at which point it returns. There are two paths for handling the data:
- IF the value is < 32 then it is a control value. Only CLS, AT and INK are recognised.
- ELSE the value is assumed to be an ASCII character and is displayed.
Whatever is happening, happens at the coordinates given by (McursorX, McursorY) which start out as (0,0) and get updated automatically when a character is output, or in response to an AT command. INK will set the required colour in Mcolour, which again starts out as 0. This is applied after the character is written to the screen, using the PRattr function.
There is a fun bit of optimisation going on in Mcontrol. At the start it pushes the address of the MESS function on the stack, which means that the RET will jump back to the start of MESS rather than where the jump happened to Mcontrol itself.
There is another shortcut in the Mcls function: LDIR. From http://z80-heaven.wikidot.com/instructions-set:ldir: “Repeats LDI (LD (DE),(HL), then increments DE, HL, and decrements BC) until BC=0.” By setting the contents of HL (the first byte of the display) to zero, this will tile that same value across the display memory until BC, which starts at $1AFF, is zero. This will zero the whole display – both pixels and attributes – from 0x4000 through to 0x5AFF.
Now finally, we get to the main update loop.
LOOP:
HALT
CALL UPDATE ; Update the display from the current Sound parameters
LD A,2
OUT (254),A ; Set border to 2
CALL REFRESH ; Update the sound driver parameters
XOR A
OUT (254),A ; Set border to 0
CALL CLOCK ; Run 50Hz clock
CALL KEYSCAN ; Guess what - scans the keyboard 🙂
LD A,07FH
IN A,(254) ; Reads 0x7FFE which is the bottom row of the keyboard
AND 1
JP NZ,LOOP ; Checks bit 0, which is the SPACE key
LD BC,65533 ; AY OUTPUT PORTS (FFFD, BFFD)
LD A,7
OUT (C),A
LD BC,49149
LD A,63 ; Set AY register 7 to 63 - i.e. all channels OFF
OUT (C),A
POP HL
POP DE
POP BC
POP AF
RETI’m not going through the sub routines of the loop, other than to note the following:
- UPDATE is a whole series of instructions that basically do the following to output the HEX value of a sound parameter:
LD A, (contents of one of the sound variables)
LD DE, (corresponding screen address for the variable to be displayed)
CALL HEX- REFRESH runs the sound driver itself, as described in Z80 and AY-3-8910.
- CLOCK decrements the FIFTY variable and every time it gets to zero updates SECS and MINS and writes them out to the display. As it also uses the HEX routine, I guess it is storing the time using binary-coded decimal (BCD).
- KEYSCAN reads the last key pressed from the system variable location stored in ASCII (23560 / 0x5C08).
At some point I might come back and work out what keys do what…
Closing Thoughts
I’d really like to get some of this code running on some of the alternate Z80 platforms I have. Getting the sound output shouldn’t be too much of an issue, but I’d really like to have some kind of display too.
But as can be seen above, the tester UI is pretty well tied into the oddities of the ZX Spectrum display, so porting it won’t be trivial.
I suspect there are already some existing AY/chiptune players that perhaps would be a better starting point, but from what I’ve seen they tend to stream the register data after having sampled it at regular intervals, which isn’t quite what I was after… there would be something really quite interesting about actually running Ste Ruddy’s Sound Driver with a Tim Follin soundtrack programmed in.
Kevin
#ay38910 #TimFollin #zxSpectrum -
No, not him, the dessert.
"While executing a flawless Baked Alaska is still an impressive feat today, back when it was initially created, it would have been even more noteworthy. There were no electric ice cream machines, stand mixers to whip up flawless meringue, or blowtorches to effortlessly toast the exterior. It was a time-consuming treat that required many hands on deck to create all the components. While you can combine any flavors of ice cream and cake to create a Baked Alaska, the original version featured bananas, which were considered quite expensive and exotic at the time, as they had to be imported from Central America.
To reflect the effort and exotic ingredients required for the dessert, the original Delmonico's price tag would have been the equivalent of about $40 today. Though you can try it for about $24 now, you'll still get the same flavor experience as a diner in the Gilded Age, as the restaurant still uses the same classic flavor combination of walnut cake, apricot jam and banana gelato.
The dessert menu of many restaurants nowadays is a lot simpler, and the idea of a dessert that requires such precise timing and such a theatrical tableside presentation has fallen out of favor. However, it remains a unique sweet offering. While you're searching for it on a menu near you, we have the recipe for a mint version so you can whip one up at home."
https://www.mashed.com/1915735/once-popular-70s-cake-baked-alaska-people-dont-eat-today/
"Like sex trafficking panic more broadly, the Epstein files are a useful political tool—as long as they remain hidden.
Remember QAnon? The conspiracy theory, popular among President Donald Trump's supporters for years, held that a furtive network of satanic pedophiles had infiltrated political institutions, the press, and other echelons of power and influence. According to Q lore, Trump was working to expose and punish these cretins—who included Hillary Clinton and other prominent Democrats—and to end the child suffering, sex trafficking, and hormone harvesting that these horrible globalists were engaging in.
QAnon—itself preceded by the perhaps even more zany Pizzagate conspiracy—was too out there for mainstream Republicans to explicitly endorse. But neither would they explicitly condemn the theory or its believers, and they often seemed happy to play along with the narrative, touting Trump's alleged crackdown on child sex trafficking.
Republicans took a lot of flak for this, but of course, Democrats had been stoking sex trafficking panic for their own gain, too. QAnon didn't start in a vacuum; it followed two decades of mainstream media and politicians, both left and right, using the flimsiest evidence and most distorted data to falsely claim that America was in the midst of a sex trafficking epidemic."
https://reason.com/2025/07/16/magas-epstein-files-fight-shows-the-long-tail-of-qanon/
-
**Проблема: Ограниченная доступность децентрализованных поисковых решений на основе YaCy в Gentoo**
**1. Децентрализация vs Централизованные поисковики**
Большинство пользователей привыкли к централизованным поисковым системам (Google, Bing, Yandex), которые контролируют индексацию, фильтрацию контента и ранжирование. YaCy предлагает децентрализованный подход, но его популярность остаётся низкой из-за ряда технических и пользовательских барьеров.
**2. Проблемы установки и совместимости в Gentoo**
Gentoo известен своей гибкостью, но установка YaCy на этой платформе может быть сложной из-за:
Отсутствия официального ebuild в основном репозитории.
Потенциальных зависимостей, конфликтующих с текущими сборками.
Отсутствия подробной документации для интеграции с системой.
**3. Ограниченная функциональность и удобство для конечного пользователя**
Хотя YaCy мощен с точки зрения приватности и автономности, он сталкивается с проблемами:
Высокие требования к ресурсам при индексировании.
Медленная скорость поиска при малом числе узлов.
Ограниченные механизмы фильтрации контента по сравнению с традиционными поисковиками.
**4. Интеграция в экосистему RuTracker.org**
На форумах вроде RuTracker.org востребованы альтернативные поисковые решения. Однако:
YaCy не всегда эффективно индексирует динамический контент форумов.
Необходима дополнительная настройка парсеров для корректного сбора данных.
Малое количество узлов, ориентированных на индексирование RuTracker, снижает качество поиска.
**Вывод**
YaCy в Gentoo и его потенциальная интеграция с RuTracker.org требуют более удобных инструментов развертывания, оптимизации индексации и повышения удобства работы для конечных пользователей.
**Дополнительная база знаний для изучения и исследования**
**1. Официальные ресурсы YaCy**
Официальный сайт YaCy – документация, исходный код и последние обновления.
GitHub YaCy – основной репозиторий проекта, баг-трекер, pull requests.
Форум поддержки YaCy – обсуждения, вопросы и ответы.
**2. Документация и исследования по децентрализованным поисковикам**
DHT (Distributed Hash Table) и его применение в P2P-системах
Сравнение децентрализованных поисковиков: SearX, YaCy, Whoogle
Peer-to-Peer Search Engines: Opportunities and Challenges (ACM Digital Library)
**3. Gentoo и его экосистема**
Официальная документация Gentoo – руководство по установке и настройке пакетов.
Bugzilla Gentoo – поиск и обсуждение ошибок, возможное добавление ebuild для YaCy.
GURU overlay – сообщество разработчиков, вносящих новые пакеты.**Библиография**
Callan, J. (2000). *Distributed Information Retrieval*. Springer.
Balakrishnan, H., Kaashoek, M. F., Karger, D., Morris, R., & Stoica, I. (2003). *Looking up data in P2P systems*. Communications of the ACM, 46(2), 43-48.
Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., & Balakrishnan, H. (2001). *Chord: A scalable peer-to-peer lookup service for internet applications*. ACM SIGCOMM Computer Communication Review, 31(4), 149-160.
Benzmüller, C., & Heyer, G. (2008). *Peer-to-peer information retrieval: An overview*. Springer.
Gentoo Linux Wiki (2024). *Installing and Configuring Packages in Gentoo*. Retrieved from https://wiki.gentoo.org.
YaCy Developers (2023). *YaCy Search Engine: Architecture and Performance Optimization*. Retrieved from https://github.com/yacy/yacy_search_server.**Хэштеги**
#YaCy #DecentralizedSearch #Gentoo #RuTracker #P2P #DistributedSearch #DHT #FOSS #PrivacyTech #PeerToPeer #OpenSource**Где найти соратников для обсуждения?**
🔹 **Официальные сообщества YaCy**
Форум YaCy Community
Группа в Matrix: #yacy:matrix.org
IRC-канал: #yacy на irc.libera.chat
🔹 **Сообщества по Gentoo и Open Source**
Форум Gentoo
Reddit: r/Gentoo
Telegram-группа Gentoo Russia
🔹 **Дискуссионные площадки по децентрализованным технологиям**
LOR (Linux.org.ru) – обсуждение Linux и open-source решений.
RuTracker.org – форум альтернативных технологий
Hacker News – обсуждение перспектив P2P и децентрализованных систем.
Эти ресурсы помогут разработчикам, исследователям и энтузиастам YaCy глубже разобраться в технологии и найти единомышленников. -
Head’s up: This is a blog post about applied cryptography, with a focus on web and cloud applications that encrypt data at rest in a database or filesystem. While the lessons can be broadly applicable, the scope of the post is not.
One of the lessons I learned during my time at AWS Cryptography (and particularly as an AWS Crypto Bar Raiser) is that the threat model for Encryption At Rest is often undefined.
Prior to consulting cryptography experts, most software developers do not have a clear and concise understanding of the risks they’re facing, let alone how or why the encrypting data at rest would help protect their customers.
Unsurprisingly, I’ve heard a few infosec thought leader types insist that encryption-at-rest is security theater over the years. I disagree with this assessment in the absolute terms, but there is a nugget of truth in that assertion.
The million dollar question.Let’s explore this subject in a little more detail.
Why should we listen to you about this topic?
(If you don’t need any convincing, feel free to skip this section.)
Encryption at rest is a particular hobby horse of mine. I previously wrote on this blog about the under-celebrated design decisions in the AWS Database Encryption SDK and the need for key-committing AEAD modes in multi-tenant data lakes.
Before my time at Amazon, I had also designed a PHP library called CipherSweet that offers a limited type of Searchable Encryption. The goal of CipherSweet was to improve the cryptography used by SuiteCRM. (The library name is, of course, a pun.)
I’ve also contributed a ton of time making cryptography easy-to-use and hard to misuse outside of the narrow use-case that is at-rest data encryption. To that end, I designed PASETO as a secure-by-default alternative to JSON Web Tokens.
I also have a lot of skin in the game when it comes to developer comprehension: I was the first Stack Overflow user with a gold badge for both [security] and [encryption], largely due to the effort I put into cleaning up the bad cryptography advice for the PHP ecosystem.
I have spent the past decade or so trying to help teams avoid security disasters in one form or another.
Why should we not listen to you about this topic?
If you happen to know a cryptography expert you trust more than some Internet stranger with a blog, I implore you to listen to them if we disagree on any point. They may know something I don’t. (That said, I’m always happy to learn something new!)
I also do not have a college degree in Cryptography, nor have I published any papers in prestigious academic journals. If you care very much about this sort of pedigree, you will likely find my words easily discarded. If this describes your situation, no hard feelings.
Why and How to use Encryption At Rest to Protect Sensitive Data
Important: I’m chiefly interested in discussing one use-case, and not focusing on other use cases. Namely, I’m focusing on encryption-at-rest in the narrow context of web applications and/or cloud services.
This is not a comprehensive blog post covering every possible use case or threat model relating to encryption at rest. Those other use cases are certainly interesting, but this post is already long enough with a narrower focus.
In particular: I’m not talking about the threats faced by activists or whistleblowers. This is a software engineering and applied cryptography focused blog post.
If you’re only interested in compliance requirements, you can probably just enable Full Disk Encryption and call it a day. Then, if your server’s hard drive grows legs and walks out of the data center, your users’ most sensitive data will remain confidential.
Unfortunately, for the server-side encryption at rest use case, that’s basically all that Disk Encryption protects against.
If your application or database software is online and an attacker gains access to it (e.g., through SQL injection), with full disk encryption, it might as well be plaintext to an online attacker.
It do be like that with online attacks.Therefore, if you find yourself reaching for Encryption At Rest to mitigate the impact of the kind of vulnerability that would leak the contents of your database or filesystem to an attacker, you’re probably unwittingly engaging in security theater.
Disk Encryption is important for disk disposal and mitigating hardware theft, not preventing data leakage to online attackers.
So the next logical thing to do is draw a box around the system or component that stores a lot of data and never let plaintext cross that boundary.
What Do You Mean By “Encryption At Rest”?
Encryption At Rest is best contrasted with Encrypted In Transit.
For Encryption-in-Transit, think TLS.
For Encryption-at-Rest, think of anything a web app or cloud service would do to encrypt data before storing it in… where ever the data is actually stored.
If there’s another usage of the term to mean something else, it’s not one that I’m familiar with.
Client-Side Encryption
Note: The naming here is a little imprecise. It is client-side encryption with respect to your data warehouse (i.e. SQL database), but not with respect to the user experience of a web application. In those cases, client-side would mean on the actual end user’s device.
Instead, client-side encryption is the generic buzz-word to mean that you’re encrypting data outside of the box you drew in your system architecture. Generally, this means that you have an application server that’s acting as the “client” for the purpose of bulk data encryption.
There are a lot of software projects that aim to provide client-side encryption for data stored in a database or filesystems; e.g., in Amazon S3 buckets.
This is a step in the right direction, but implementation details matter a lot.
Quick aside: For the remainder of this blog post, I’m going to assume an architecture that looks like a traditional web application, for simplicity.
The assumed architecture looks vaguely like this:
- User Agents (e.g., web browsers) that communicate with the application server.
- Application Server(s) respond to HTTP requests from user agents, manages key material using KMS, encrypts / decrypts records stored in the database.
- Database Server(s) which store ciphertext on behalf of the application server.
This is an abstract design, so the actual implementation details you encounter in the real world may be simpler or more complex in different respects.
There are other interesting design considerations for OS-level end-user device encryption that I’m not going to explore today. For example: Adiantum is extremely cool.
I’m also not going to dive deep into laptop theft or the importance of Full Disk Encryption as a mechanism for ensuring data is erased from solid state hard drives, or the activities of hostile nation states. That’s a separate discussion entirely.
Security Considerations for Client-Side Encryption
The first question to answer when data is being encrypted is, “How are the keys being managed?” This is a very deep rabbit hole of complexity, but one good answer for a centralized service is, “Cloud-based key management service with audit logging”; i.e. AWS KMS, Google CloudKMS, etc.
We could talk about key management for a very long time, but there’s other things I want to focus on, so let’s revisit that in a future blog post.
Before we begin, you may find it helpful to read my previous blog post on a related matter: Lucid Multi-Key Deputies Require Commitment. It’s not strictly necessary, but some of the terminology used there may be helpful to understanding this one.
Next, you have to understand how the data is being encrypted in the first place.
Bulk Data Encryption Techniques
Bad answer: AES in CBC mode without HMAC.
Worse answer: AES in ECB mode.
Generally, you’re going to want to use an AEAD construction, such as AES-GCM or XChaCha20-Poly1305.
For those not in the loop: AEAD is an acronym that stands for Authenticated Encryption with Associated Data.
You’ll also want key-commitment if you’re storing data for multiple customers in the same hardware. You can get this property by stapling HKDF onto your protocol (once for key derivation, again for commitment). See also: PASETO v3 and v4, or Version 2 of the AWS Encryption SDK.
It may be tempting to build your own custom committing AEAD scheme out of, e.g., AES-CTR and HMAC. If you do this, take extra care that you don’t introduce canonicalization risks in your MAC.
Either way, using an AEAD mode is a significant improvement over using AES directly.
Is Your Deputy Confused?
Even if you’re using IND-CCA secure encryption and managing your keys securely, there is still a very stupid attack against many data-at-rest encryption schemes.
To understand the attack, first consider this sort of scenario:
Alice and Bob use the same health insurance provider, who is storing sensitive medical records for both parties. Bob works as a database administrator for the insurance company he and Alice both use. One day, he decides to snoop on her private medical history.
Fortunately, the data is encrypted at the web application, so all of the data Bob can access is indistinguishable from random. He can access his own account and see his data through the application, but he cannot see Alice’s data from his vantage point on the database server.
Here’s the stupid simple attack that works in far too many cases: Bob copies Alice’s encrypted data, and overwrites his records in the database, then accesses the insurance provider’s web app.
Bam! Alice’s plaintext recovered.
What’s happening here is simple: The web application has the ability to decrypt different records encrypted with different keys. If you pass records that were encrypted for Alice to the application to decrypt it for Bob, and you’re not authenticating your access patterns, Bob can read Alice’s data by performing this attack.
The cryptographic attack is literally copy and paste, from the database administrator’s perspective. It’s stupid but it works against too many encryption-at-rest software projects.
In this setup, the application is the Deputy, and you can easily confuse it by replaying an encrypted blob in the incorrect context.
The mitigation is simple: Use the AAD mechanism (part of the standard AEAD interface) to bind a ciphertext to its context. This can be a customer ID, each row’s value for the primary key of the database table, or something else entirely.
If you’re using AWS KMS, you can also use Encryption Context for this exact purpose.
An Illustrative Example
Let’s say you have a simple web application that encrypts data before storing it in a SQL database.
Let’s also write it to use AES-GCM, since unauthenticated CBC mode is awful.
A quick and dirty implementation might look like this:
class User { public function __construct( public readonly string $username, public string $email, public string $fullName ) {}}class UserModel { public function __construct(protected Database $db) {} public function save(User $user): bool { return $this->db->upsert( 'users', [ // set 'full_name' => aes128gcm_encrypt($user->fullName), 'email' => aes128gcm_encrypt($user->email) // encryption details abstracted ], [ // where 'username' => $user->username ] ); } public function fetch(string $username): User { $row = $this->db->fetch('users', ['username' => $username]); return new User( $username, aes128gcm_decrypt($row['email']), aes128gcm_decrypt($row['full_name']) ); }}For the abstracted
aes128gcmfunctions in the pseudocode above, just assume they’re getting the key from KMS during encryption and storing an encrypted data key in a place the ciphertext can reference later on decrypt. I didn’t want to complicate the pseudocode with a lot of boilerplate.You might decide to prove the confused deputy risk by doing something like this:
$model = new UserModel($db);$model->save(new User('alice', '[email protected]', 'Alice McWonderland'));$model->save(new User('bob', '[email protected]', 'Bob BurgerMeister'));// Fetch Alice's data$aliceData = $db->fetch('users', ['username' => 'alice']);$bobData = $db->fetch('users', ['username' => 'bob']);// This is the attack the database server can perfrom:// Replace Bob's full_name with Alice's email$db->upsert('users', [ 'full_name' => $alice['email']], ['username' => 'bob']);$badBob = $model->fetch('bob');Now Bob’s full name is set to Alice’s email address.
Okay, So What?
Now imagine someone performs the same attack, but against salary fields in a payroll system.
The Curious Case of CipherSweet
My knowledge of this risk didn’t manifest itself in a vacuum. It was discovered over the years of maintaining an open source library.
The first release of CipherSweet mitigated most of this risk by construction: Each field uses a different encryption key, through a key derivation scheme.
In pseudocode, this construction looks something like this:
def encryptRow(self, records): for field, type in self.fieldsToEncrypt: key = self.getFieldSymmetricKey(self.table, field) records[field] = encryptField(key, field)
Since CipherSweet’s inception, if you try to replace Alice’s encrypted zip code with Alice’s encrypted social security number, the keys would be wrong, so it would lead to a decryption failure.
Or so I thought!
As I mentioned in my blog post about multi-tenancy and confused deputy attacks, if your AEAD mode doesn’t commit to the key used, it’s possible to craft a single (ciphertext, tag) that decrypts to two different plaintext values under two different keys.
CipherSweet’s original
ModernCryptosuite used XChaCha20-Poly1305, which is not key-committing, and therefore susceptible to this sort of misuse.This violated the Principle of Least Astonishment and motivated the development of a new algorithm suite called
BoringCrypto, which used BLAKE2b-MAC instead of Poly1305. This change was released in version 3.0.0 in June 2021.However, even with
BoringCryptoin 3.0.0, this only mitigated most of the issue by construction. The last mile of complexity here is that each field must also be bound to a primary key or foreign key.Encrypting with AAD has been possible since a very early release of CipherSweet, but being possible to use securely is not sufficient. It should be easy to use securely.
CipherSweet Version 4.7.0, which was released last month, now only requires a code change that looks like this in order to mitigate confused deputies in an application:
$multiRowEncryptor = new EncryptedMultiRows($engine); $multiRowEncryptor+ ->setAutoBindContext(true)+ ->setPrimaryKeyColumn('table2', 'id') ->addTextField('table1', 'field1')This is in addition to the new Enhanced AAD feature, which allows for flexible and powerful context binding based on other fields and/or string literals.
(In fact, this new convenience feature actually uses Enhanced AAD under-the-hood.)
This doesn’t come for free, however: Users have to know the serial / primary key for a record prior to writing it, in order to use it as AAD when encrypting fields. However, that’s a much easier pill to swallow than expecting PHP devs to manage the complexity of context-binding themselves.
As you can see, mitigating confused deputies in an encryption library (without making it unwieldy) requires a painstaking attention to detail to get right.
As Avi Douglen says, “Security at the cost of usability comes at the cost of security.”
Given the prevalence of client-side encryption projects that just phone it in with insecure block cipher modes (or ECB, which is the absence of a block cipher mode entirely), it’s highly doubtful that most of them will ever address confused deputy attacks. Even I didn’t get it right at first when I made CipherSweet back in 2018.
What about non-databases?
Everything I mentioned in the previous section was focused on confused deputy attacks against client-side encryption for information that is stored in a database, but it’s a general problem with encrypting data at rest and storing the ciphertext “server-side”.
If you’re storing encrypted data in an S3 bucket, rather than in MySQL, you still need some form of context-binding mechanism to prevent the dumb and obvious attack from working against a deputy that reads data from said S3 bucket.
If you take nothing else away from this blog post, remember: Authenticate your access patterns.
Why aren’t things better already?
As with most things in software security, the problem is either not widely known, or is not widely understood.
Unknown unknowns tend to fester, untreated, across the entire ecosystem.
Misunderstood issues often lead to an incorrect solution.
In this case, at-rest encryption is mostly in Column B, and confused deputy attacks are mostly in Column A.
The most pronounced consequence of this is, when tasked with building at-rest data encryption in an application, most software developers do not have a cohesive threat model in mind (let alone a formal one).
This leads to disagreement between stakeholders about what the security requirements actually are.
How can I help improve things somewhat?
Most importantly, spread awareness of the nuances of encryption at-rest.
This blog post is intended to be a good conversation starter, but there are other resources to consider, too. I’ve linked to many of them throughout this post already.
If you’re paying for software to encrypt data at rest, ask your vendor how they mitigate the risk of confused deputy attacks. Link them to this blog post if they’re not sure what you mean.
If said vendor responds, “this risk is outside of our threat model,” ask to see their formal threat model document. If it exists and doesn’t align with your application’s threat model, maybe consider alternative solutions that provide protection against more attack classes than Full Disk Encryption would.
Finally, gaining experience with threat modeling is a good use of every developer’s time. Adam Caudill has an excellent introductory blog post on the subject.
Closing Thoughts
Despite everything I’ve written here today, I do not claim to have all the answers for encryption at rest.
However, you can unlock a lot of value just by asking the right questions. My hope is that anyone that reads this post is now capable of asking those questions.
Addendum (2024-06-03)
After I published this, the r/netsec subreddit has expressed disappointment that this blog post had “no mention of” consumer device theft or countries experiencing civil unrest and pulling hard drives from data centers.
You could make a congruent complaint that it also had no mention of Batman.
To be clear, I’m not saying that the use cases and risks Reddit cares about are off-topic to any discussion of full-disk encryption. They matter.
Rather, it’s that they’re not relevant to the specific point I am making: Even in the simplest use case, far from the annoying details of end user hardware or the whims of nation states, encryption-at-rest is poorly understood by most developers, and should be thought through carefully.
Your threat model is not my threat model, and vice versa.
I never advertised this blog post as a comprehensive and complete guide to the entire subject of encryption-at-rest. If you too felt under-served by this blog post for not addressing the corner cases that really matter to you, I hope this addendum makes it clearer why I didn’t cover them.
Finally, if you feel that there’s an aspect of the encryption-at-rest topic that really warrants further examination, I invite you to blog about it.
If your blog post is interesting enough, I’ll revise this post and link to it here.
https://scottarc.blog/2024/06/02/encryption-at-rest-whose-threat-model-is-it-anyway/
#Cryptography #cybersecurity #encryption #encryptionAtRest #security #symmetricCryptography #technology
-
Hacker News Front Page Analytics ... what next?
I'm thinking through where else to take this. I've had a few side discussions and commentary here and at HN. Part of this is coming up with questions, part with the tools to answer them.
The initial question concerned places and regional references found on the HN front page. My initial analysis answered that (US states), and it was pretty easy to add cities (US and global) and countries to the list.
I also wanted some overall summary statistics, for all time, by year, by period (I've done weekdays, I still need to get to months). There were some interesting comparisons --- vote and comment activity by page position, for example (there's an 844.4 point advantage for 1st over 30th place in votes, 340.3 in comments, for 2022, on average).
I've broken out overall and average (per-story) votes and comments, which is interesting.
There's top-site and top-user activity, and how that changes over time. I've done some work on this, I'm thinking of both other questions and how to represent this graphically.
(Graphical representation is a question for other aspects as well ... what I've created so far is great for people who like reading 100s of pages of tables, less for those who prefer a visual representation.)
What I've done less of, and am trying to think of ways to surface interesting elements rather than be strictly query/question driven, is to find patterns and trends in the data itself, most especially in the title text. There are challenges: HN doesn't provide much to work with (titles are restricted to 80 characters, generally), and there is of course ambiguity, though I'd posted a set of interesting/amusing items (see: https://toot.cat/@dredmorbius/110454128168815763).
I've been playing with some simple ngram code (awk associative arrays of 2..5 elements ... mind-bogglingly easy to create and often surprisingly insightful).
I've relied on some external lists of entities (states, cities, countries, etc.) which are useful. I'd done an earlier analysis based on the Foreign Policy Top 100 Global Thinkers list, assessing salience level of various online sources, in 2015 (see: https://old.reddit.com/r/dredmorbius/comments/3hp41w/tracking_the_conversation_fp_global_100_thinkers/). I can re-use that list, though I'd like to find a few others --- top startups / companies / people. Also perhaps major stories and terms from the past two decades. (I've done some searches based on my own recollection, e.g., MeToo, BLM, George Floyd, and the like with some success).
And I'd like to do a deeper parse of the source HTML to grab both HN threads and source URLs. I've found the html-xml-utils package useful, need to check that's installed locally (OK, seems it is) and wrap my head around it again (the tools are ... idiosyncratic). Oh, and homebrew lists package executables in /usr/local/opt//bin/, which is good to know. Yay!
(Yes, I'm aware there are other tools. I'm a simple basher.)
-
**Проблема: Ограниченная доступность децентрализованных поисковых решений на основе YaCy в Gentoo**
**1. Децентрализация vs Централизованные поисковики**
Большинство пользователей привыкли к централизованным поисковым системам (Google, Bing, Yandex), которые контролируют индексацию, фильтрацию контента и ранжирование. YaCy предлагает децентрализованный подход, но его популярность остаётся низкой из-за ряда технических и пользовательских барьеров.
**2. Проблемы установки и совместимости в Gentoo**
Gentoo известен своей гибкостью, но установка YaCy на этой платформе может быть сложной из-за:
Отсутствия официального ebuild в основном репозитории.
Потенциальных зависимостей, конфликтующих с текущими сборками.
Отсутствия подробной документации для интеграции с системой.
**3. Ограниченная функциональность и удобство для конечного пользователя**
Хотя YaCy мощен с точки зрения приватности и автономности, он сталкивается с проблемами:
Высокие требования к ресурсам при индексировании.
Медленная скорость поиска при малом числе узлов.
Ограниченные механизмы фильтрации контента по сравнению с традиционными поисковиками.
**4. Интеграция в экосистему RuTracker.org**
На форумах вроде RuTracker.org востребованы альтернативные поисковые решения. Однако:
YaCy не всегда эффективно индексирует динамический контент форумов.
Необходима дополнительная настройка парсеров для корректного сбора данных.
Малое количество узлов, ориентированных на индексирование RuTracker, снижает качество поиска.
**Вывод**
YaCy в Gentoo и его потенциальная интеграция с RuTracker.org требуют более удобных инструментов развертывания, оптимизации индексации и повышения удобства работы для конечных пользователей.
**Дополнительная база знаний для изучения и исследования**
**1. Официальные ресурсы YaCy**
Официальный сайт YaCy – документация, исходный код и последние обновления.
GitHub YaCy – основной репозиторий проекта, баг-трекер, pull requests.
Форум поддержки YaCy – обсуждения, вопросы и ответы.
**2. Документация и исследования по децентрализованным поисковикам**
DHT (Distributed Hash Table) и его применение в P2P-системах
Сравнение децентрализованных поисковиков: SearX, YaCy, Whoogle
Peer-to-Peer Search Engines: Opportunities and Challenges (ACM Digital Library)
**3. Gentoo и его экосистема**
Официальная документация Gentoo – руководство по установке и настройке пакетов.
Bugzilla Gentoo – поиск и обсуждение ошибок, возможное добавление ebuild для YaCy.
GURU overlay – сообщество разработчиков, вносящих новые пакеты.**Библиография**
Callan, J. (2000). *Distributed Information Retrieval*. Springer.
Balakrishnan, H., Kaashoek, M. F., Karger, D., Morris, R., & Stoica, I. (2003). *Looking up data in P2P systems*. Communications of the ACM, 46(2), 43-48.
Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., & Balakrishnan, H. (2001). *Chord: A scalable peer-to-peer lookup service for internet applications*. ACM SIGCOMM Computer Communication Review, 31(4), 149-160.
Benzmüller, C., & Heyer, G. (2008). *Peer-to-peer information retrieval: An overview*. Springer.
Gentoo Linux Wiki (2024). *Installing and Configuring Packages in Gentoo*. Retrieved from https://wiki.gentoo.org.
YaCy Developers (2023). *YaCy Search Engine: Architecture and Performance Optimization*. Retrieved from https://github.com/yacy/yacy_search_server.**Хэштеги**
#YaCy #DecentralizedSearch #Gentoo #RuTracker #P2P #DistributedSearch #DHT #FOSS #PrivacyTech #PeerToPeer #OpenSource**Где найти соратников для обсуждения?**
🔹 **Официальные сообщества YaCy**
Форум YaCy Community
Группа в Matrix: #yacy:matrix.org
IRC-канал: #yacy на irc.libera.chat
🔹 **Сообщества по Gentoo и Open Source**
Форум Gentoo
Reddit: r/Gentoo
Telegram-группа Gentoo Russia
🔹 **Дискуссионные площадки по децентрализованным технологиям**
LOR (Linux.org.ru) – обсуждение Linux и open-source решений.
RuTracker.org – форум альтернативных технологий
Hacker News – обсуждение перспектив P2P и децентрализованных систем.
Эти ресурсы помогут разработчикам, исследователям и энтузиастам YaCy глубже разобраться в технологии и найти единомышленников. -
**Проблема: Ограниченная доступность децентрализованных поисковых решений на основе YaCy в Gentoo**
**1. Децентрализация vs Централизованные поисковики**
Большинство пользователей привыкли к централизованным поисковым системам (Google, Bing, Yandex), которые контролируют индексацию, фильтрацию контента и ранжирование. YaCy предлагает децентрализованный подход, но его популярность остаётся низкой из-за ряда технических и пользовательских барьеров.
**2. Проблемы установки и совместимости в Gentoo**
Gentoo известен своей гибкостью, но установка YaCy на этой платформе может быть сложной из-за:
Отсутствия официального ebuild в основном репозитории.
Потенциальных зависимостей, конфликтующих с текущими сборками.
Отсутствия подробной документации для интеграции с системой.
**3. Ограниченная функциональность и удобство для конечного пользователя**
Хотя YaCy мощен с точки зрения приватности и автономности, он сталкивается с проблемами:
Высокие требования к ресурсам при индексировании.
Медленная скорость поиска при малом числе узлов.
Ограниченные механизмы фильтрации контента по сравнению с традиционными поисковиками.
**4. Интеграция в экосистему RuTracker.org**
На форумах вроде RuTracker.org востребованы альтернативные поисковые решения. Однако:
YaCy не всегда эффективно индексирует динамический контент форумов.
Необходима дополнительная настройка парсеров для корректного сбора данных.
Малое количество узлов, ориентированных на индексирование RuTracker, снижает качество поиска.
**Вывод**
YaCy в Gentoo и его потенциальная интеграция с RuTracker.org требуют более удобных инструментов развертывания, оптимизации индексации и повышения удобства работы для конечных пользователей.
**Дополнительная база знаний для изучения и исследования**
**1. Официальные ресурсы YaCy**
Официальный сайт YaCy – документация, исходный код и последние обновления.
GitHub YaCy – основной репозиторий проекта, баг-трекер, pull requests.
Форум поддержки YaCy – обсуждения, вопросы и ответы.
**2. Документация и исследования по децентрализованным поисковикам**
DHT (Distributed Hash Table) и его применение в P2P-системах
Сравнение децентрализованных поисковиков: SearX, YaCy, Whoogle
Peer-to-Peer Search Engines: Opportunities and Challenges (ACM Digital Library)
**3. Gentoo и его экосистема**
Официальная документация Gentoo – руководство по установке и настройке пакетов.
Bugzilla Gentoo – поиск и обсуждение ошибок, возможное добавление ebuild для YaCy.
GURU overlay – сообщество разработчиков, вносящих новые пакеты.**Библиография**
Callan, J. (2000). *Distributed Information Retrieval*. Springer.
Balakrishnan, H., Kaashoek, M. F., Karger, D., Morris, R., & Stoica, I. (2003). *Looking up data in P2P systems*. Communications of the ACM, 46(2), 43-48.
Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., & Balakrishnan, H. (2001). *Chord: A scalable peer-to-peer lookup service for internet applications*. ACM SIGCOMM Computer Communication Review, 31(4), 149-160.
Benzmüller, C., & Heyer, G. (2008). *Peer-to-peer information retrieval: An overview*. Springer.
Gentoo Linux Wiki (2024). *Installing and Configuring Packages in Gentoo*. Retrieved from https://wiki.gentoo.org.
YaCy Developers (2023). *YaCy Search Engine: Architecture and Performance Optimization*. Retrieved from https://github.com/yacy/yacy_search_server.**Хэштеги**
#YaCy #DecentralizedSearch #Gentoo #RuTracker #P2P #DistributedSearch #DHT #FOSS #PrivacyTech #PeerToPeer #OpenSource**Где найти соратников для обсуждения?**
🔹 **Официальные сообщества YaCy**
Форум YaCy Community
Группа в Matrix: #yacy:matrix.org
IRC-канал: #yacy на irc.libera.chat
🔹 **Сообщества по Gentoo и Open Source**
Форум Gentoo
Reddit: r/Gentoo
Telegram-группа Gentoo Russia
🔹 **Дискуссионные площадки по децентрализованным технологиям**
LOR (Linux.org.ru) – обсуждение Linux и open-source решений.
RuTracker.org – форум альтернативных технологий
Hacker News – обсуждение перспектив P2P и децентрализованных систем.
Эти ресурсы помогут разработчикам, исследователям и энтузиастам YaCy глубже разобраться в технологии и найти единомышленников. -
**Проблема: Ограниченная доступность децентрализованных поисковых решений на основе YaCy в Gentoo**
**1. Децентрализация vs Централизованные поисковики**
Большинство пользователей привыкли к централизованным поисковым системам (Google, Bing, Yandex), которые контролируют индексацию, фильтрацию контента и ранжирование. YaCy предлагает децентрализованный подход, но его популярность остаётся низкой из-за ряда технических и пользовательских барьеров.
**2. Проблемы установки и совместимости в Gentoo**
Gentoo известен своей гибкостью, но установка YaCy на этой платформе может быть сложной из-за:
Отсутствия официального ebuild в основном репозитории.
Потенциальных зависимостей, конфликтующих с текущими сборками.
Отсутствия подробной документации для интеграции с системой.
**3. Ограниченная функциональность и удобство для конечного пользователя**
Хотя YaCy мощен с точки зрения приватности и автономности, он сталкивается с проблемами:
Высокие требования к ресурсам при индексировании.
Медленная скорость поиска при малом числе узлов.
Ограниченные механизмы фильтрации контента по сравнению с традиционными поисковиками.
**4. Интеграция в экосистему RuTracker.org**
На форумах вроде RuTracker.org востребованы альтернативные поисковые решения. Однако:
YaCy не всегда эффективно индексирует динамический контент форумов.
Необходима дополнительная настройка парсеров для корректного сбора данных.
Малое количество узлов, ориентированных на индексирование RuTracker, снижает качество поиска.
**Вывод**
YaCy в Gentoo и его потенциальная интеграция с RuTracker.org требуют более удобных инструментов развертывания, оптимизации индексации и повышения удобства работы для конечных пользователей.
**Дополнительная база знаний для изучения и исследования**
**1. Официальные ресурсы YaCy**
Официальный сайт YaCy – документация, исходный код и последние обновления.
GitHub YaCy – основной репозиторий проекта, баг-трекер, pull requests.
Форум поддержки YaCy – обсуждения, вопросы и ответы.
**2. Документация и исследования по децентрализованным поисковикам**
DHT (Distributed Hash Table) и его применение в P2P-системах
Сравнение децентрализованных поисковиков: SearX, YaCy, Whoogle
Peer-to-Peer Search Engines: Opportunities and Challenges (ACM Digital Library)
**3. Gentoo и его экосистема**
Официальная документация Gentoo – руководство по установке и настройке пакетов.
Bugzilla Gentoo – поиск и обсуждение ошибок, возможное добавление ebuild для YaCy.
GURU overlay – сообщество разработчиков, вносящих новые пакеты.**Библиография**
Callan, J. (2000). *Distributed Information Retrieval*. Springer.
Balakrishnan, H., Kaashoek, M. F., Karger, D., Morris, R., & Stoica, I. (2003). *Looking up data in P2P systems*. Communications of the ACM, 46(2), 43-48.
Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., & Balakrishnan, H. (2001). *Chord: A scalable peer-to-peer lookup service for internet applications*. ACM SIGCOMM Computer Communication Review, 31(4), 149-160.
Benzmüller, C., & Heyer, G. (2008). *Peer-to-peer information retrieval: An overview*. Springer.
Gentoo Linux Wiki (2024). *Installing and Configuring Packages in Gentoo*. Retrieved from https://wiki.gentoo.org.
YaCy Developers (2023). *YaCy Search Engine: Architecture and Performance Optimization*. Retrieved from https://github.com/yacy/yacy_search_server.**Хэштеги**
#YaCy #DecentralizedSearch #Gentoo #RuTracker #P2P #DistributedSearch #DHT #FOSS #PrivacyTech #PeerToPeer #OpenSource**Где найти соратников для обсуждения?**
🔹 **Официальные сообщества YaCy**
Форум YaCy Community
Группа в Matrix: #yacy:matrix.org
IRC-канал: #yacy на irc.libera.chat
🔹 **Сообщества по Gentoo и Open Source**
Форум Gentoo
Reddit: r/Gentoo
Telegram-группа Gentoo Russia
🔹 **Дискуссионные площадки по децентрализованным технологиям**
LOR (Linux.org.ru) – обсуждение Linux и open-source решений.
RuTracker.org – форум альтернативных технологий
Hacker News – обсуждение перспектив P2P и децентрализованных систем.
Эти ресурсы помогут разработчикам, исследователям и энтузиастам YaCy глубже разобраться в технологии и найти единомышленников. -
**Проблема: Ограниченная доступность децентрализованных поисковых решений на основе YaCy в Gentoo**
**1. Децентрализация vs Централизованные поисковики**
Большинство пользователей привыкли к централизованным поисковым системам (Google, Bing, Yandex), которые контролируют индексацию, фильтрацию контента и ранжирование. YaCy предлагает децентрализованный подход, но его популярность остаётся низкой из-за ряда технических и пользовательских барьеров.
**2. Проблемы установки и совместимости в Gentoo**
Gentoo известен своей гибкостью, но установка YaCy на этой платформе может быть сложной из-за:
Отсутствия официального ebuild в основном репозитории.
Потенциальных зависимостей, конфликтующих с текущими сборками.
Отсутствия подробной документации для интеграции с системой.
**3. Ограниченная функциональность и удобство для конечного пользователя**
Хотя YaCy мощен с точки зрения приватности и автономности, он сталкивается с проблемами:
Высокие требования к ресурсам при индексировании.
Медленная скорость поиска при малом числе узлов.
Ограниченные механизмы фильтрации контента по сравнению с традиционными поисковиками.
**4. Интеграция в экосистему RuTracker.org**
На форумах вроде RuTracker.org востребованы альтернативные поисковые решения. Однако:
YaCy не всегда эффективно индексирует динамический контент форумов.
Необходима дополнительная настройка парсеров для корректного сбора данных.
Малое количество узлов, ориентированных на индексирование RuTracker, снижает качество поиска.
**Вывод**
YaCy в Gentoo и его потенциальная интеграция с RuTracker.org требуют более удобных инструментов развертывания, оптимизации индексации и повышения удобства работы для конечных пользователей.
**Дополнительная база знаний для изучения и исследования**
**1. Официальные ресурсы YaCy**
Официальный сайт YaCy – документация, исходный код и последние обновления.
GitHub YaCy – основной репозиторий проекта, баг-трекер, pull requests.
Форум поддержки YaCy – обсуждения, вопросы и ответы.
**2. Документация и исследования по децентрализованным поисковикам**
DHT (Distributed Hash Table) и его применение в P2P-системах
Сравнение децентрализованных поисковиков: SearX, YaCy, Whoogle
Peer-to-Peer Search Engines: Opportunities and Challenges (ACM Digital Library)
**3. Gentoo и его экосистема**
Официальная документация Gentoo – руководство по установке и настройке пакетов.
Bugzilla Gentoo – поиск и обсуждение ошибок, возможное добавление ebuild для YaCy.
GURU overlay – сообщество разработчиков, вносящих новые пакеты.**Библиография**
Callan, J. (2000). *Distributed Information Retrieval*. Springer.
Balakrishnan, H., Kaashoek, M. F., Karger, D., Morris, R., & Stoica, I. (2003). *Looking up data in P2P systems*. Communications of the ACM, 46(2), 43-48.
Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., & Balakrishnan, H. (2001). *Chord: A scalable peer-to-peer lookup service for internet applications*. ACM SIGCOMM Computer Communication Review, 31(4), 149-160.
Benzmüller, C., & Heyer, G. (2008). *Peer-to-peer information retrieval: An overview*. Springer.
Gentoo Linux Wiki (2024). *Installing and Configuring Packages in Gentoo*. Retrieved from https://wiki.gentoo.org.
YaCy Developers (2023). *YaCy Search Engine: Architecture and Performance Optimization*. Retrieved from https://github.com/yacy/yacy_search_server.**Хэштеги**
#YaCy #DecentralizedSearch #Gentoo #RuTracker #P2P #DistributedSearch #DHT #FOSS #PrivacyTech #PeerToPeer #OpenSource**Где найти соратников для обсуждения?**
🔹 **Официальные сообщества YaCy**
Форум YaCy Community
Группа в Matrix: #yacy:matrix.org
IRC-канал: #yacy на irc.libera.chat
🔹 **Сообщества по Gentoo и Open Source**
Форум Gentoo
Reddit: r/Gentoo
Telegram-группа Gentoo Russia
🔹 **Дискуссионные площадки по децентрализованным технологиям**
LOR (Linux.org.ru) – обсуждение Linux и open-source решений.
RuTracker.org – форум альтернативных технологий
Hacker News – обсуждение перспектив P2P и децентрализованных систем.
Эти ресурсы помогут разработчикам, исследователям и энтузиастам YaCy глубже разобраться в технологии и найти единомышленников. -
Cannabis Lies Vol. 9: The Reform Lie
Filed Under: Policy Fiction
The federal apparatus has spoken. The Department of Justice and the Drug Enforcement Administration have announced a shift in the regulatory status of cannabis, moving state-licensed medical products to Schedule III under the Controlled Substances Act while pointedly leaving adult use, unlicensed, and synthetic THC products in Schedule I. Headlines across the country erupted with the language of victory. Outlets hailed this as a historic acknowledgment of the plant’s medical utility, a shift that supposedly recognizes the plant’s reality after decades of denial. The public was told that the prohibition era was entering its twilight and that the federal government had finally conceded that the plant possesses medicinal value.
None of this reflects the actual legal impact of the order. This announcement is the latest manifestation of the Reform Lie. It is a calculated piece of bureaucratic maintenance designed to satisfy the demand for progress while ensuring the core structure of prohibition remains entirely untouched. As Acting US Attorney General Todd Blanche stated in the order, the new policy mandates that:
“Marijuana in any form covered by a state medical marijuana license, be placed in Schedule III of the Controlled Substances Act.”
It is a classic maneuver by the state to preserve its authority by offering a small, controlled concession that changes everything on paper but leaves the reality of the drug war exactly where it has always been.
The Reform Lie is the mechanism by which the state manages the tension between popular opinion and its own mandate. It functions by acknowledging that a substance has medical value without ever addressing the fundamental injustice of its criminalization. When the government moves a substance from one box to another, it claims it is following the science. When that same government keeps the prisons full, keeps the borders militarized against possession, and keeps the threat of federal intervention hanging over every state-sanctioned interaction, it is not following science. It is managing optics. For decades, the apparatus has faced growing pressure to address the disconnect between federal law and the public reality of cannabis use. Instead of dismantling the structure, the government has repeatedly opted for symbolic reform. These gestures generate cycles of positive press. They allow officials to claim they have acted on the issue. They provide a release valve for public anger without ever sacrificing the underlying authority to arrest, prosecute, and punish. This is the central trick. The Reform Lie presents a change in tax status as a change in morality.
To understand the scope of this deception, one must look closely at what the shift to Schedule III actually achieves. Under the Controlled Substances Act, Schedule III is home to substances such as anabolic steroids and certain prescription painkillers. It is a designation that implies a potential for abuse, though one that the state deems less severe than those in the Schedule I category, which the government defines as having no currently accepted medical use. Moving state-licensed medical products to Schedule III finally acknowledges what has been true for thousands of years. It acknowledges that the plant has medical value.
However, the change in classification does nothing to address the core conflicts of the prohibition era. The federal criminal penalties for the unauthorized production, distribution, or possession of cannabis remain firmly in place for everything outside that narrow, state-sanctioned medical window. The interstate commerce ban survives completely intact. The government continues to treat the transport of the plant across state lines as a federal crime, regardless of the legality of the substance in the states of origin or destination. Banking remains a fractured landscape of private risk and federal oversight. Employment in the federal sector remains hostile to users, and the firearm restrictions that strip rights from medical patients do not budge.
Most critically, this move provides no relief for those currently held in the carceral system. This order structurally excludes any mechanism for record relief, sentence modification, or pardon, leaving the carceral status quo entirely intact. It does not vacate criminal records. It does not end the status of cannabis as a tool for immigration enforcement. It does not stop the random, localized harassment of the population by federal agencies that still view the plant as contraband outside of the narrow, state-licensed framework.
This is a victory for the balance sheet. It is a win for the corporations that have spent millions lobbying for the ability to deduct ordinary business expenses under the tax code. As of April 22, 2026, state-licensed medical cannabis is no longer subject to 280E. It is a stabilization for the industry that the government has deemed acceptable. For the average person, for the patient, and for the citizen who does not operate within the protective bubble of a state-licensed medical program, the reality remains frozen in the past. This bifurcation of the population is intentional. It creates a system where legitimacy is not a right inherent to the citizen. It is a commodity to be licensed. The people who work within the sanctioned industry are protected, taxed, and monitored. The people who exist outside of that bubble, who grow their own, who share, or who live in states without functional medical programs, are left to the mercy of a law that has not changed. The government has not legalized the plant. It has simply professionalized the privilege of interacting with it.
This strategy is not new. It follows a consistent historical pattern. In every generation, the state has used cannabis policy as a messaging tool to address shifting cultural demands. This is not about the plant. It is about the maintenance of control. The lineage of this deception is long and well-documented.
Consider the era of the Gateway Lie. The government needed a way to justify the expansion of its police power, so it framed the plant as the first step on a path to hard drug use. This narrative was never about safety. It was about creating a bridge between a benign cultural habit and the perceived chaos of the heroin epidemic. It gave law enforcement a justification to monitor, harass, and incarcerate individuals who were otherwise peaceful. The Gateway Lie was effective because it operated on fear. It suggested that a single act of consumption was a moral failing that would lead inevitably to destruction.
Consider the Crime Lie, where the plant was the supposed accelerant for violence. In the 1980s and 1990s, the state pivoted to a narrative of aggression. It claimed that cannabis use caused psychosis and fueled the drug trade. It used this narrative to justify the militarization of police forces, the introduction of civil asset forfeiture, and the explosion of the prison population. The Crime Lie turned the consumer into a danger to the public, a threat that had to be neutralized by the full weight of the judicial system. It was never about the drug. It was about the expansion of the carceral state.
Consider the Teen Epidemic Lie, where the narrative focused on the alleged destruction of youth, or the Addiction Lie, which served to pathologize a human relationship with a plant. Each of these lies served a purpose. They provided the state with the moral cover required to expand surveillance, increase budgets, and exert control. The Reform Lie is simply the modern evolution of this pattern. The state no longer needs to argue that the plant causes violence, because the public no longer believes it. So, the state shifts the narrative. It pivots to the language of regulation. It claims to be fixing the system. It is a retreat, but it is a managed retreat. The goal remains the same, which is to maintain the state’s position as the final arbiter of what a person can put into their own body.
The most devastating impact of the Reform Lie is the erasure of the human cost. When the headlines celebrate a minor technical shift, they drown out the voices of those who continue to suffer under the full weight of prohibition. The Reform Lie tells the prisoner that their incarceration is necessary because they did not have the right paperwork. It tells the immigrant that their status remains precarious because the federal law still views the plant as an illicit substance. It tells the veteran that they must choose between their medical treatment and their access to federal services. It tells the small grower that they are a criminal while the corporate entity next door is a taxpayer. By focusing on the tax status of corporations, the conversation ignores the individuals who are still being processed through the system. It creates an environment where progress is measured by market capitalization rather than the restoration of liberty. It turns the struggle for sovereignty into a fight for market share.
Help Keep Pot Culture Magazine Independent Pot Culture Magazine is independent cannabis journalism. No corporate owners. No investors. Just readers. If you value this work, chip in a few dollars and help keep it going. Support PCMIf the government acknowledges that cannabis has medical value, the continued maintenance of criminal penalties for everyone else becomes an indefensible moral contradiction. One cannot simultaneously argue that a substance is legitimate medicine and that the possession of that substance warrants the stripping of rights, the loss of employment, or the threat of prison. This contradiction exposes the truth of the state position. The government does not actually care about the safety of the substance. It cares about the control of the substance. If it were about safety, the state would be looking for ways to educate rather than incarcerate. If it were about medicine, the state would be ensuring access rather than creating barriers. The existence of the prohibition machinery alongside the admission of medical utility for the licensed few is proof that the objective has always been to maintain a system of punishment.
This system relies on the compliance of the public. It relies on the belief that the state is making progress. The Reform Lie is designed to prevent the public from seeing that the state is not moving toward freedom. It is moving toward an integrated model of control. By allowing a portion of the market to become legitimate, the state creates a vested interest in the status quo. The corporate entities that now have a seat at the table are no longer incentivized to fight for total legalization. They are incentivized to maintain the current regulatory structure because it keeps their competitors out. They become partners in the enforcement of the very prohibition they once railed against. This is the ultimate victory for the state. It co-opts the opposition by giving them a slice of the profit.
We have seen this happen in other sectors of the economy, where regulations are written by the very corporations they are meant to govern. This is not reform. This is the capture of the regulatory apparatus. The Reform Lie ensures that the people who built the culture, who fought for the plant when it was dangerous to do so, are excluded from the new order. They are the ones who bear the cost of the transition. They are the ones who are still in cages, who are still fleeing from the law, who are still fighting for the right to exist in peace.
This administrative process is now set to continue with new hearings starting June 29, 2026. These proceedings are often portrayed as a necessary step toward further reform, a way to build a bureaucratic consensus for future changes. In practice, they serve as a stalling tactic. They provide a way for the administrative state to maintain the illusion of progress while keeping the ultimate authority firmly in its own hands. These hearings will involve experts, lobbyists, and officials debating the minutiae of regulation, all while the fundamental structure of the Controlled Substances Act remains unassailable. The system is designed to consume time, resources, and energy, ensuring that any real change is mediated through a process that the state can control, slow, or halt entirely. It is a theatre of governance, performed for an audience that is desperate for change, but the script was written in the halls of power, not by the people who have lived the consequences of prohibition.
MORE FROM CANNABIS LIES
CANNABIS LIES Vol. 8: The Addiction Lie
Cannabis is often labeled addictive, but the science tells a more precise story. This piece breaks down cannabis use disorder, how it is defined, and why mild, moderate, and severe cases get flattened into one fear-driven narrative. The result is a distorted public understanding of risk that fuels policy, perception, and misinformation.
by Pot Culture Magazine EditorsApril 11, 2026April 20, 2026CANNABIS LIES Vol. 7: The Mental Health Panic
Cannabis and mental health risks are often overstated in public debate. Research shows heavy use and high THC exposure can increase psychosis risk in vulnerable individuals, but widespread claims of a mental health crisis lack strong evidence. This piece examines the data, separates correlation from causation, and breaks down what cannabis users need to know.
by Pot Culture Magazine EditorsApril 4, 2026April 2, 2026CANNABIS LIES Vol. 6: The Driving Apocalypse Lie
Legal cannabis is often blamed for rising traffic deaths, but federal data tells a more complicated story. NHTSA findings, toxicology limitations, and conflicting crash studies reveal that THC presence is not a reliable measure of impairment. This investigation breaks down how flawed testing and policy shortcuts have shaped the narrative around so-called stoned driving.
by Pot Culture Magazine EditorsMarch 28, 2026March 27, 2026The administrative state is also moving to consolidate its control over clinical trials. By creating a registration pathway for state-licensed entities, the government is essentially seizing control of the research process. It is setting itself up as the gatekeeper of scientific knowledge. It will dictate who can research the plant, what they can research, and what the results can be used for. This is not an opening of the doors to scientific discovery. It is the enclosure of the scientific commons. It ensures that the research that reaches the public will be the research that has been filtered through the priorities of the state.
The Reform Lie is not a strategy. It is an admission of failure. When the government chooses to perform the act of reform without embracing the reality of justice, it proves that it is not interested in the truth. It is interested in the maintenance of power. True reform would not be a shuffling of schedules. It would be the total and unconditional withdrawal of federal interference from the lives of the people. It would be the recognition that the state has no authority to criminalize the relationship between a human being and a plant. It would be the end of the prohibition machine, the release of the prisoners, and the restoration of rights for every person affected by the war on the plant.
As long as the apparatus continues to frame these technical shifts as moral victories, the public must recognize the deception. This is not progress. This is the state recalibrating its control to ensure that it remains the gatekeeper, the tax collector, and the final judge of who is allowed to exist in the world it seeks to dominate. The plant remains the same. The people remain the same. The only thing that has shifted is the label on the cage. The cage is still there. The bars are still locked. The guards are still watching. The power to punish, to threaten, and to control has not been removed. It has been refined. It has been made more surgical. It has been made more efficient.
The moral weight of this lie is heavy. It falls on those who have been promised justice and received only a change in terminology. It falls on the families who have been broken by the enforcement of archaic laws. It falls on the communities that have been targeted for generations. The Reform Lie assumes that the public has forgotten the history of the struggle. It assumes that the public is satisfied with the crumbs of corporate legitimacy. It assumes that there is no understanding of the difference between the freedom to live and the permission to serve.
The narrative of the state must be rejected. The recognition must grow that every small step that leaves the core structure of the prohibition machine in place is a step away from justice. The government must be held accountable for the contradiction of its own law. The reality of the prohibition era must continue to be documented, to expose the lies that are told to justify the control, and to advocate for the total restoration of liberty. The struggle for the plant is not a struggle for a change in status. It is a struggle for the soul of the culture. It is a struggle to define what it means to be a free person in a society that seeks to regulate every choice. As NORML Deputy Director Paul Armentano noted regarding the order:
“Rescheduling fails to fully harmonize federal marijuana policy with the cannabis laws of many states, particularly the 24 states that have legalized its use and sale to adults.”
This is the core of the deception. The Reform Lie is the latest barrier to that freedom. It is a wall that must be dismantled, not by the government, but by the people who have lived the reality of the struggle.
The truth is simple, though the state works hard to obscure it. Cannabis is a part of the human experience. It has been used for healing, for creativity, for connection, and for joy for as long as historical records exist. The attempts by the state to control this relationship are an affront to human autonomy. They are based on fear, on ignorance, and on a desire for power. The reclassification to Schedule III is just the latest tactic in a long campaign to prevent people from fully embracing their own sovereignty. While the proponents of this move claim that:
“Today’s order marks a historical reversal in federal cannabis policy,”
It is a sign that the state is feeling the pressure, that it knows its position is untenable, but that it is not yet ready to concede.
A crossroads has been reached. Either the crumbs offered by the state are accepted, turning the public into participants in their own regulation, or the fight for the total and unconditional end of the prohibition machine continues. The Reform Lie can be accepted, or the truth can be demanded. The history of the culture is a history of resistance. It is a history of people who refused to be told what they could do, who they could be, or what they could consume. That history is the source of strength. It is the foundation upon which the future will be built. Permission from the state is not required to exist. Schedules, labels, and tax codes are not needed to define what is right. The truth is known, and it will continue to be shared until the last cage is empty and the prohibition machine is nothing but a memory.
The Reform Lie will continue to be told. The headlines will continue to scream about progress that does not exist. The state will continue to frame its maintenance of power as a move toward justice. But the deception will not hold. The patterns are visible. The history is known. The stakes are understood. The reality of the prohibition era will be documented, one article, one story, one voice at a time. This is not just a battle for a plant. It is a battle for the truth. And it is a battle that will be won, not because the state gives permission, but because the truth is on the side of the people. The prohibition machine is built on lies, and lies cannot stand forever against the weight of reality. The end of prohibition is coming, not through the actions of the state, but through the resolve of the people who have been fighting for it all along. The Reform Lie is the last gasp of a system that knows its time is over. We will not be fooled. We will not be silenced. We will be here, documenting the reality, telling the truth, and fighting for the culture until the day the plant is free.
©2026, Pot Culture Magazine. All rights reserved. This is the property of Pot Culture Magazine and is protected by U.S. and international copyright laws. Unauthorized reproduction, distribution, or transmission
of this work, in part or in whole, without the express written permission of Pot Culture Magazine, is strictly
prohibited.F O R T H E C U L T U R E B Y T H E C U L T U R E
The Digital Cage: Saint Lucia’s Traceability Trap
Saint Lucia has selected GrowerIQ as its national seed-to-sale traceability backbone, effectively finalizing a digital surveillance grid for its cannabis industry. By mandating enterprise software before establishing licensing frameworks, the government risks automating the exclusion of legacy farmers. This move trades cultural sovereignty for state-managed control, turning the cannabis industry into an extension of the…
by Pot Culture Magazine EditorsApril 23, 2026April 22, 2026Gov. Abigail Spanberger’s Virginia Sabotage
Virginia legalized possession, but Governor Abigail Spanberger sabotaged the retail market. By delaying sales until 2027 and gutting equity provisions, the Commonwealth institutionalized a half-legal trap. Consumers now navigate a system that treats possession as a right but supply as a crime, fueling an unchecked illicit market while abandoning promised reform. Spanberger’s public safety rhetoric…
by Pot Culture Magazine EditorsApril 21, 2026April 20, 20264/20 has been hollowed out by branding, corporate silence, and a culture that forgot its own history. While the industry sells holiday merch, Singapore executed a man for cannabis. The movement that once fought for autonomy now treats the plant like a commodity. This piece examines the cost of that betrayal and the culture left…
by Pot Culture Magazine EditorsApril 20, 2026April 24, 2026 #280E #AdministrativeLaw #cannabis #CannabisCommunity #CannabisCulture #CannabisCommunity #CarceralState #Culture #DEA #DepartmentOfJustice #DrugWar #FederalGovernment #Industry #Legalization #Marijuana #MarijuanaNews #NORML #Policy #PolicyFiction #PotCultureMagazine #Prohibition #Reform #ScheduleIII #StateSanctioned #Weed -
Cannabis Lies Vol. 9: The Reform Lie
Filed Under: Policy Fiction
The federal apparatus has spoken. The Department of Justice and the Drug Enforcement Administration have announced a shift in the regulatory status of cannabis, moving state-licensed medical products to Schedule III under the Controlled Substances Act while pointedly leaving adult use, unlicensed, and synthetic THC products in Schedule I. Headlines across the country erupted with the language of victory. Outlets hailed this as a historic acknowledgment of the plant’s medical utility, a shift that supposedly recognizes the plant’s reality after decades of denial. The public was told that the prohibition era was entering its twilight and that the federal government had finally conceded that the plant possesses medicinal value.
None of this reflects the actual legal impact of the order. This announcement is the latest manifestation of the Reform Lie. It is a calculated piece of bureaucratic maintenance designed to satisfy the demand for progress while ensuring the core structure of prohibition remains entirely untouched. As Acting US Attorney General Todd Blanche stated in the order, the new policy mandates that:
“Marijuana in any form covered by a state medical marijuana license, be placed in Schedule III of the Controlled Substances Act.”
It is a classic maneuver by the state to preserve its authority by offering a small, controlled concession that changes everything on paper but leaves the reality of the drug war exactly where it has always been.
The Reform Lie is the mechanism by which the state manages the tension between popular opinion and its own mandate. It functions by acknowledging that a substance has medical value without ever addressing the fundamental injustice of its criminalization. When the government moves a substance from one box to another, it claims it is following the science. When that same government keeps the prisons full, keeps the borders militarized against possession, and keeps the threat of federal intervention hanging over every state-sanctioned interaction, it is not following science. It is managing optics. For decades, the apparatus has faced growing pressure to address the disconnect between federal law and the public reality of cannabis use. Instead of dismantling the structure, the government has repeatedly opted for symbolic reform. These gestures generate cycles of positive press. They allow officials to claim they have acted on the issue. They provide a release valve for public anger without ever sacrificing the underlying authority to arrest, prosecute, and punish. This is the central trick. The Reform Lie presents a change in tax status as a change in morality.
To understand the scope of this deception, one must look closely at what the shift to Schedule III actually achieves. Under the Controlled Substances Act, Schedule III is home to substances such as anabolic steroids and certain prescription painkillers. It is a designation that implies a potential for abuse, though one that the state deems less severe than those in the Schedule I category, which the government defines as having no currently accepted medical use. Moving state-licensed medical products to Schedule III finally acknowledges what has been true for thousands of years. It acknowledges that the plant has medical value.
However, the change in classification does nothing to address the core conflicts of the prohibition era. The federal criminal penalties for the unauthorized production, distribution, or possession of cannabis remain firmly in place for everything outside that narrow, state-sanctioned medical window. The interstate commerce ban survives completely intact. The government continues to treat the transport of the plant across state lines as a federal crime, regardless of the legality of the substance in the states of origin or destination. Banking remains a fractured landscape of private risk and federal oversight. Employment in the federal sector remains hostile to users, and the firearm restrictions that strip rights from medical patients do not budge.
Most critically, this move provides no relief for those currently held in the carceral system. This order structurally excludes any mechanism for record relief, sentence modification, or pardon, leaving the carceral status quo entirely intact. It does not vacate criminal records. It does not end the status of cannabis as a tool for immigration enforcement. It does not stop the random, localized harassment of the population by federal agencies that still view the plant as contraband outside of the narrow, state-licensed framework.
This is a victory for the balance sheet. It is a win for the corporations that have spent millions lobbying for the ability to deduct ordinary business expenses under the tax code. As of April 22, 2026, state-licensed medical cannabis is no longer subject to 280E. It is a stabilization for the industry that the government has deemed acceptable. For the average person, for the patient, and for the citizen who does not operate within the protective bubble of a state-licensed medical program, the reality remains frozen in the past. This bifurcation of the population is intentional. It creates a system where legitimacy is not a right inherent to the citizen. It is a commodity to be licensed. The people who work within the sanctioned industry are protected, taxed, and monitored. The people who exist outside of that bubble, who grow their own, who share, or who live in states without functional medical programs, are left to the mercy of a law that has not changed. The government has not legalized the plant. It has simply professionalized the privilege of interacting with it.
This strategy is not new. It follows a consistent historical pattern. In every generation, the state has used cannabis policy as a messaging tool to address shifting cultural demands. This is not about the plant. It is about the maintenance of control. The lineage of this deception is long and well-documented.
Consider the era of the Gateway Lie. The government needed a way to justify the expansion of its police power, so it framed the plant as the first step on a path to hard drug use. This narrative was never about safety. It was about creating a bridge between a benign cultural habit and the perceived chaos of the heroin epidemic. It gave law enforcement a justification to monitor, harass, and incarcerate individuals who were otherwise peaceful. The Gateway Lie was effective because it operated on fear. It suggested that a single act of consumption was a moral failing that would lead inevitably to destruction.
Consider the Crime Lie, where the plant was the supposed accelerant for violence. In the 1980s and 1990s, the state pivoted to a narrative of aggression. It claimed that cannabis use caused psychosis and fueled the drug trade. It used this narrative to justify the militarization of police forces, the introduction of civil asset forfeiture, and the explosion of the prison population. The Crime Lie turned the consumer into a danger to the public, a threat that had to be neutralized by the full weight of the judicial system. It was never about the drug. It was about the expansion of the carceral state.
Consider the Teen Epidemic Lie, where the narrative focused on the alleged destruction of youth, or the Addiction Lie, which served to pathologize a human relationship with a plant. Each of these lies served a purpose. They provided the state with the moral cover required to expand surveillance, increase budgets, and exert control. The Reform Lie is simply the modern evolution of this pattern. The state no longer needs to argue that the plant causes violence, because the public no longer believes it. So, the state shifts the narrative. It pivots to the language of regulation. It claims to be fixing the system. It is a retreat, but it is a managed retreat. The goal remains the same, which is to maintain the state’s position as the final arbiter of what a person can put into their own body.
The most devastating impact of the Reform Lie is the erasure of the human cost. When the headlines celebrate a minor technical shift, they drown out the voices of those who continue to suffer under the full weight of prohibition. The Reform Lie tells the prisoner that their incarceration is necessary because they did not have the right paperwork. It tells the immigrant that their status remains precarious because the federal law still views the plant as an illicit substance. It tells the veteran that they must choose between their medical treatment and their access to federal services. It tells the small grower that they are a criminal while the corporate entity next door is a taxpayer. By focusing on the tax status of corporations, the conversation ignores the individuals who are still being processed through the system. It creates an environment where progress is measured by market capitalization rather than the restoration of liberty. It turns the struggle for sovereignty into a fight for market share.
Help Keep Pot Culture Magazine Independent Pot Culture Magazine is independent cannabis journalism. No corporate owners. No investors. Just readers. If you value this work, chip in a few dollars and help keep it going. Support PCMIf the government acknowledges that cannabis has medical value, the continued maintenance of criminal penalties for everyone else becomes an indefensible moral contradiction. One cannot simultaneously argue that a substance is legitimate medicine and that the possession of that substance warrants the stripping of rights, the loss of employment, or the threat of prison. This contradiction exposes the truth of the state position. The government does not actually care about the safety of the substance. It cares about the control of the substance. If it were about safety, the state would be looking for ways to educate rather than incarcerate. If it were about medicine, the state would be ensuring access rather than creating barriers. The existence of the prohibition machinery alongside the admission of medical utility for the licensed few is proof that the objective has always been to maintain a system of punishment.
This system relies on the compliance of the public. It relies on the belief that the state is making progress. The Reform Lie is designed to prevent the public from seeing that the state is not moving toward freedom. It is moving toward an integrated model of control. By allowing a portion of the market to become legitimate, the state creates a vested interest in the status quo. The corporate entities that now have a seat at the table are no longer incentivized to fight for total legalization. They are incentivized to maintain the current regulatory structure because it keeps their competitors out. They become partners in the enforcement of the very prohibition they once railed against. This is the ultimate victory for the state. It co-opts the opposition by giving them a slice of the profit.
We have seen this happen in other sectors of the economy, where regulations are written by the very corporations they are meant to govern. This is not reform. This is the capture of the regulatory apparatus. The Reform Lie ensures that the people who built the culture, who fought for the plant when it was dangerous to do so, are excluded from the new order. They are the ones who bear the cost of the transition. They are the ones who are still in cages, who are still fleeing from the law, who are still fighting for the right to exist in peace.
This administrative process is now set to continue with new hearings starting June 29, 2026. These proceedings are often portrayed as a necessary step toward further reform, a way to build a bureaucratic consensus for future changes. In practice, they serve as a stalling tactic. They provide a way for the administrative state to maintain the illusion of progress while keeping the ultimate authority firmly in its own hands. These hearings will involve experts, lobbyists, and officials debating the minutiae of regulation, all while the fundamental structure of the Controlled Substances Act remains unassailable. The system is designed to consume time, resources, and energy, ensuring that any real change is mediated through a process that the state can control, slow, or halt entirely. It is a theatre of governance, performed for an audience that is desperate for change, but the script was written in the halls of power, not by the people who have lived the consequences of prohibition.
MORE FROM CANNABIS LIES
CANNABIS LIES Vol. 8: The Addiction Lie
Cannabis is often labeled addictive, but the science tells a more precise story. This piece breaks down cannabis use disorder, how it is defined, and why mild, moderate, and severe cases get flattened into one fear-driven narrative. The result is a distorted public understanding of risk that fuels policy, perception, and misinformation.
by Pot Culture Magazine EditorsApril 11, 2026April 20, 2026CANNABIS LIES Vol. 7: The Mental Health Panic
Cannabis and mental health risks are often overstated in public debate. Research shows heavy use and high THC exposure can increase psychosis risk in vulnerable individuals, but widespread claims of a mental health crisis lack strong evidence. This piece examines the data, separates correlation from causation, and breaks down what cannabis users need to know.
by Pot Culture Magazine EditorsApril 4, 2026April 2, 2026CANNABIS LIES Vol. 6: The Driving Apocalypse Lie
Legal cannabis is often blamed for rising traffic deaths, but federal data tells a more complicated story. NHTSA findings, toxicology limitations, and conflicting crash studies reveal that THC presence is not a reliable measure of impairment. This investigation breaks down how flawed testing and policy shortcuts have shaped the narrative around so-called stoned driving.
by Pot Culture Magazine EditorsMarch 28, 2026March 27, 2026The administrative state is also moving to consolidate its control over clinical trials. By creating a registration pathway for state-licensed entities, the government is essentially seizing control of the research process. It is setting itself up as the gatekeeper of scientific knowledge. It will dictate who can research the plant, what they can research, and what the results can be used for. This is not an opening of the doors to scientific discovery. It is the enclosure of the scientific commons. It ensures that the research that reaches the public will be the research that has been filtered through the priorities of the state.
The Reform Lie is not a strategy. It is an admission of failure. When the government chooses to perform the act of reform without embracing the reality of justice, it proves that it is not interested in the truth. It is interested in the maintenance of power. True reform would not be a shuffling of schedules. It would be the total and unconditional withdrawal of federal interference from the lives of the people. It would be the recognition that the state has no authority to criminalize the relationship between a human being and a plant. It would be the end of the prohibition machine, the release of the prisoners, and the restoration of rights for every person affected by the war on the plant.
As long as the apparatus continues to frame these technical shifts as moral victories, the public must recognize the deception. This is not progress. This is the state recalibrating its control to ensure that it remains the gatekeeper, the tax collector, and the final judge of who is allowed to exist in the world it seeks to dominate. The plant remains the same. The people remain the same. The only thing that has shifted is the label on the cage. The cage is still there. The bars are still locked. The guards are still watching. The power to punish, to threaten, and to control has not been removed. It has been refined. It has been made more surgical. It has been made more efficient.
The moral weight of this lie is heavy. It falls on those who have been promised justice and received only a change in terminology. It falls on the families who have been broken by the enforcement of archaic laws. It falls on the communities that have been targeted for generations. The Reform Lie assumes that the public has forgotten the history of the struggle. It assumes that the public is satisfied with the crumbs of corporate legitimacy. It assumes that there is no understanding of the difference between the freedom to live and the permission to serve.
The narrative of the state must be rejected. The recognition must grow that every small step that leaves the core structure of the prohibition machine in place is a step away from justice. The government must be held accountable for the contradiction of its own law. The reality of the prohibition era must continue to be documented, to expose the lies that are told to justify the control, and to advocate for the total restoration of liberty. The struggle for the plant is not a struggle for a change in status. It is a struggle for the soul of the culture. It is a struggle to define what it means to be a free person in a society that seeks to regulate every choice. As NORML Deputy Director Paul Armentano noted regarding the order:
“Rescheduling fails to fully harmonize federal marijuana policy with the cannabis laws of many states, particularly the 24 states that have legalized its use and sale to adults.”
This is the core of the deception. The Reform Lie is the latest barrier to that freedom. It is a wall that must be dismantled, not by the government, but by the people who have lived the reality of the struggle.
The truth is simple, though the state works hard to obscure it. Cannabis is a part of the human experience. It has been used for healing, for creativity, for connection, and for joy for as long as historical records exist. The attempts by the state to control this relationship are an affront to human autonomy. They are based on fear, on ignorance, and on a desire for power. The reclassification to Schedule III is just the latest tactic in a long campaign to prevent people from fully embracing their own sovereignty. While the proponents of this move claim that:
“Today’s order marks a historical reversal in federal cannabis policy,”
It is a sign that the state is feeling the pressure, that it knows its position is untenable, but that it is not yet ready to concede.
A crossroads has been reached. Either the crumbs offered by the state are accepted, turning the public into participants in their own regulation, or the fight for the total and unconditional end of the prohibition machine continues. The Reform Lie can be accepted, or the truth can be demanded. The history of the culture is a history of resistance. It is a history of people who refused to be told what they could do, who they could be, or what they could consume. That history is the source of strength. It is the foundation upon which the future will be built. Permission from the state is not required to exist. Schedules, labels, and tax codes are not needed to define what is right. The truth is known, and it will continue to be shared until the last cage is empty and the prohibition machine is nothing but a memory.
The Reform Lie will continue to be told. The headlines will continue to scream about progress that does not exist. The state will continue to frame its maintenance of power as a move toward justice. But the deception will not hold. The patterns are visible. The history is known. The stakes are understood. The reality of the prohibition era will be documented, one article, one story, one voice at a time. This is not just a battle for a plant. It is a battle for the truth. And it is a battle that will be won, not because the state gives permission, but because the truth is on the side of the people. The prohibition machine is built on lies, and lies cannot stand forever against the weight of reality. The end of prohibition is coming, not through the actions of the state, but through the resolve of the people who have been fighting for it all along. The Reform Lie is the last gasp of a system that knows its time is over. We will not be fooled. We will not be silenced. We will be here, documenting the reality, telling the truth, and fighting for the culture until the day the plant is free.
©2026, Pot Culture Magazine. All rights reserved. This is the property of Pot Culture Magazine and is protected by U.S. and international copyright laws. Unauthorized reproduction, distribution, or transmission
of this work, in part or in whole, without the express written permission of Pot Culture Magazine, is strictly
prohibited.F O R T H E C U L T U R E B Y T H E C U L T U R E
The Digital Cage: Saint Lucia’s Traceability Trap
Saint Lucia has selected GrowerIQ as its national seed-to-sale traceability backbone, effectively finalizing a digital surveillance grid for its cannabis industry. By mandating enterprise software before establishing licensing frameworks, the government risks automating the exclusion of legacy farmers. This move trades cultural sovereignty for state-managed control, turning the cannabis industry into an extension of the…
by Pot Culture Magazine EditorsApril 23, 2026April 22, 2026Gov. Abigail Spanberger’s Virginia Sabotage
Virginia legalized possession, but Governor Abigail Spanberger sabotaged the retail market. By delaying sales until 2027 and gutting equity provisions, the Commonwealth institutionalized a half-legal trap. Consumers now navigate a system that treats possession as a right but supply as a crime, fueling an unchecked illicit market while abandoning promised reform. Spanberger’s public safety rhetoric…
by Pot Culture Magazine EditorsApril 21, 2026April 20, 20264/20 has been hollowed out by branding, corporate silence, and a culture that forgot its own history. While the industry sells holiday merch, Singapore executed a man for cannabis. The movement that once fought for autonomy now treats the plant like a commodity. This piece examines the cost of that betrayal and the culture left…
by Pot Culture Magazine EditorsApril 20, 2026April 24, 2026 #280E #AdministrativeLaw #cannabis #CannabisCommunity #CannabisCulture #CannabisCommunity #CarceralState #Culture #DEA #DepartmentOfJustice #DrugWar #FederalGovernment #Industry #Legalization #Marijuana #MarijuanaNews #NORML #Policy #PolicyFiction #PotCultureMagazine #Prohibition #Reform #ScheduleIII #StateSanctioned #Weed -
Cannabis Lies Vol. 9: The Reform Lie
Filed Under: Policy Fiction
The federal apparatus has spoken. The Department of Justice and the Drug Enforcement Administration have announced a shift in the regulatory status of cannabis, moving state-licensed medical products to Schedule III under the Controlled Substances Act while pointedly leaving adult use, unlicensed, and synthetic THC products in Schedule I. Headlines across the country erupted with the language of victory. Outlets hailed this as a historic acknowledgment of the plant’s medical utility, a shift that supposedly recognizes the plant’s reality after decades of denial. The public was told that the prohibition era was entering its twilight and that the federal government had finally conceded that the plant possesses medicinal value.
None of this reflects the actual legal impact of the order. This announcement is the latest manifestation of the Reform Lie. It is a calculated piece of bureaucratic maintenance designed to satisfy the demand for progress while ensuring the core structure of prohibition remains entirely untouched. As Acting US Attorney General Todd Blanche stated in the order, the new policy mandates that:
“Marijuana in any form covered by a state medical marijuana license, be placed in Schedule III of the Controlled Substances Act.”
It is a classic maneuver by the state to preserve its authority by offering a small, controlled concession that changes everything on paper but leaves the reality of the drug war exactly where it has always been.
The Reform Lie is the mechanism by which the state manages the tension between popular opinion and its own mandate. It functions by acknowledging that a substance has medical value without ever addressing the fundamental injustice of its criminalization. When the government moves a substance from one box to another, it claims it is following the science. When that same government keeps the prisons full, keeps the borders militarized against possession, and keeps the threat of federal intervention hanging over every state-sanctioned interaction, it is not following science. It is managing optics. For decades, the apparatus has faced growing pressure to address the disconnect between federal law and the public reality of cannabis use. Instead of dismantling the structure, the government has repeatedly opted for symbolic reform. These gestures generate cycles of positive press. They allow officials to claim they have acted on the issue. They provide a release valve for public anger without ever sacrificing the underlying authority to arrest, prosecute, and punish. This is the central trick. The Reform Lie presents a change in tax status as a change in morality.
To understand the scope of this deception, one must look closely at what the shift to Schedule III actually achieves. Under the Controlled Substances Act, Schedule III is home to substances such as anabolic steroids and certain prescription painkillers. It is a designation that implies a potential for abuse, though one that the state deems less severe than those in the Schedule I category, which the government defines as having no currently accepted medical use. Moving state-licensed medical products to Schedule III finally acknowledges what has been true for thousands of years. It acknowledges that the plant has medical value.
However, the change in classification does nothing to address the core conflicts of the prohibition era. The federal criminal penalties for the unauthorized production, distribution, or possession of cannabis remain firmly in place for everything outside that narrow, state-sanctioned medical window. The interstate commerce ban survives completely intact. The government continues to treat the transport of the plant across state lines as a federal crime, regardless of the legality of the substance in the states of origin or destination. Banking remains a fractured landscape of private risk and federal oversight. Employment in the federal sector remains hostile to users, and the firearm restrictions that strip rights from medical patients do not budge.
Most critically, this move provides no relief for those currently held in the carceral system. This order structurally excludes any mechanism for record relief, sentence modification, or pardon, leaving the carceral status quo entirely intact. It does not vacate criminal records. It does not end the status of cannabis as a tool for immigration enforcement. It does not stop the random, localized harassment of the population by federal agencies that still view the plant as contraband outside of the narrow, state-licensed framework.
This is a victory for the balance sheet. It is a win for the corporations that have spent millions lobbying for the ability to deduct ordinary business expenses under the tax code. As of April 22, 2026, state-licensed medical cannabis is no longer subject to 280E. It is a stabilization for the industry that the government has deemed acceptable. For the average person, for the patient, and for the citizen who does not operate within the protective bubble of a state-licensed medical program, the reality remains frozen in the past. This bifurcation of the population is intentional. It creates a system where legitimacy is not a right inherent to the citizen. It is a commodity to be licensed. The people who work within the sanctioned industry are protected, taxed, and monitored. The people who exist outside of that bubble, who grow their own, who share, or who live in states without functional medical programs, are left to the mercy of a law that has not changed. The government has not legalized the plant. It has simply professionalized the privilege of interacting with it.
This strategy is not new. It follows a consistent historical pattern. In every generation, the state has used cannabis policy as a messaging tool to address shifting cultural demands. This is not about the plant. It is about the maintenance of control. The lineage of this deception is long and well-documented.
Consider the era of the Gateway Lie. The government needed a way to justify the expansion of its police power, so it framed the plant as the first step on a path to hard drug use. This narrative was never about safety. It was about creating a bridge between a benign cultural habit and the perceived chaos of the heroin epidemic. It gave law enforcement a justification to monitor, harass, and incarcerate individuals who were otherwise peaceful. The Gateway Lie was effective because it operated on fear. It suggested that a single act of consumption was a moral failing that would lead inevitably to destruction.
Consider the Crime Lie, where the plant was the supposed accelerant for violence. In the 1980s and 1990s, the state pivoted to a narrative of aggression. It claimed that cannabis use caused psychosis and fueled the drug trade. It used this narrative to justify the militarization of police forces, the introduction of civil asset forfeiture, and the explosion of the prison population. The Crime Lie turned the consumer into a danger to the public, a threat that had to be neutralized by the full weight of the judicial system. It was never about the drug. It was about the expansion of the carceral state.
Consider the Teen Epidemic Lie, where the narrative focused on the alleged destruction of youth, or the Addiction Lie, which served to pathologize a human relationship with a plant. Each of these lies served a purpose. They provided the state with the moral cover required to expand surveillance, increase budgets, and exert control. The Reform Lie is simply the modern evolution of this pattern. The state no longer needs to argue that the plant causes violence, because the public no longer believes it. So, the state shifts the narrative. It pivots to the language of regulation. It claims to be fixing the system. It is a retreat, but it is a managed retreat. The goal remains the same, which is to maintain the state’s position as the final arbiter of what a person can put into their own body.
The most devastating impact of the Reform Lie is the erasure of the human cost. When the headlines celebrate a minor technical shift, they drown out the voices of those who continue to suffer under the full weight of prohibition. The Reform Lie tells the prisoner that their incarceration is necessary because they did not have the right paperwork. It tells the immigrant that their status remains precarious because the federal law still views the plant as an illicit substance. It tells the veteran that they must choose between their medical treatment and their access to federal services. It tells the small grower that they are a criminal while the corporate entity next door is a taxpayer. By focusing on the tax status of corporations, the conversation ignores the individuals who are still being processed through the system. It creates an environment where progress is measured by market capitalization rather than the restoration of liberty. It turns the struggle for sovereignty into a fight for market share.
Help Keep Pot Culture Magazine Independent Pot Culture Magazine is independent cannabis journalism. No corporate owners. No investors. Just readers. If you value this work, chip in a few dollars and help keep it going. Support PCMIf the government acknowledges that cannabis has medical value, the continued maintenance of criminal penalties for everyone else becomes an indefensible moral contradiction. One cannot simultaneously argue that a substance is legitimate medicine and that the possession of that substance warrants the stripping of rights, the loss of employment, or the threat of prison. This contradiction exposes the truth of the state position. The government does not actually care about the safety of the substance. It cares about the control of the substance. If it were about safety, the state would be looking for ways to educate rather than incarcerate. If it were about medicine, the state would be ensuring access rather than creating barriers. The existence of the prohibition machinery alongside the admission of medical utility for the licensed few is proof that the objective has always been to maintain a system of punishment.
This system relies on the compliance of the public. It relies on the belief that the state is making progress. The Reform Lie is designed to prevent the public from seeing that the state is not moving toward freedom. It is moving toward an integrated model of control. By allowing a portion of the market to become legitimate, the state creates a vested interest in the status quo. The corporate entities that now have a seat at the table are no longer incentivized to fight for total legalization. They are incentivized to maintain the current regulatory structure because it keeps their competitors out. They become partners in the enforcement of the very prohibition they once railed against. This is the ultimate victory for the state. It co-opts the opposition by giving them a slice of the profit.
We have seen this happen in other sectors of the economy, where regulations are written by the very corporations they are meant to govern. This is not reform. This is the capture of the regulatory apparatus. The Reform Lie ensures that the people who built the culture, who fought for the plant when it was dangerous to do so, are excluded from the new order. They are the ones who bear the cost of the transition. They are the ones who are still in cages, who are still fleeing from the law, who are still fighting for the right to exist in peace.
This administrative process is now set to continue with new hearings starting June 29, 2026. These proceedings are often portrayed as a necessary step toward further reform, a way to build a bureaucratic consensus for future changes. In practice, they serve as a stalling tactic. They provide a way for the administrative state to maintain the illusion of progress while keeping the ultimate authority firmly in its own hands. These hearings will involve experts, lobbyists, and officials debating the minutiae of regulation, all while the fundamental structure of the Controlled Substances Act remains unassailable. The system is designed to consume time, resources, and energy, ensuring that any real change is mediated through a process that the state can control, slow, or halt entirely. It is a theatre of governance, performed for an audience that is desperate for change, but the script was written in the halls of power, not by the people who have lived the consequences of prohibition.
MORE FROM CANNABIS LIES
CANNABIS LIES Vol. 8: The Addiction Lie
Cannabis is often labeled addictive, but the science tells a more precise story. This piece breaks down cannabis use disorder, how it is defined, and why mild, moderate, and severe cases get flattened into one fear-driven narrative. The result is a distorted public understanding of risk that fuels policy, perception, and misinformation.
by Pot Culture Magazine EditorsApril 11, 2026April 20, 2026CANNABIS LIES Vol. 7: The Mental Health Panic
Cannabis and mental health risks are often overstated in public debate. Research shows heavy use and high THC exposure can increase psychosis risk in vulnerable individuals, but widespread claims of a mental health crisis lack strong evidence. This piece examines the data, separates correlation from causation, and breaks down what cannabis users need to know.
by Pot Culture Magazine EditorsApril 4, 2026April 2, 2026CANNABIS LIES Vol. 6: The Driving Apocalypse Lie
Legal cannabis is often blamed for rising traffic deaths, but federal data tells a more complicated story. NHTSA findings, toxicology limitations, and conflicting crash studies reveal that THC presence is not a reliable measure of impairment. This investigation breaks down how flawed testing and policy shortcuts have shaped the narrative around so-called stoned driving.
by Pot Culture Magazine EditorsMarch 28, 2026March 27, 2026The administrative state is also moving to consolidate its control over clinical trials. By creating a registration pathway for state-licensed entities, the government is essentially seizing control of the research process. It is setting itself up as the gatekeeper of scientific knowledge. It will dictate who can research the plant, what they can research, and what the results can be used for. This is not an opening of the doors to scientific discovery. It is the enclosure of the scientific commons. It ensures that the research that reaches the public will be the research that has been filtered through the priorities of the state.
The Reform Lie is not a strategy. It is an admission of failure. When the government chooses to perform the act of reform without embracing the reality of justice, it proves that it is not interested in the truth. It is interested in the maintenance of power. True reform would not be a shuffling of schedules. It would be the total and unconditional withdrawal of federal interference from the lives of the people. It would be the recognition that the state has no authority to criminalize the relationship between a human being and a plant. It would be the end of the prohibition machine, the release of the prisoners, and the restoration of rights for every person affected by the war on the plant.
As long as the apparatus continues to frame these technical shifts as moral victories, the public must recognize the deception. This is not progress. This is the state recalibrating its control to ensure that it remains the gatekeeper, the tax collector, and the final judge of who is allowed to exist in the world it seeks to dominate. The plant remains the same. The people remain the same. The only thing that has shifted is the label on the cage. The cage is still there. The bars are still locked. The guards are still watching. The power to punish, to threaten, and to control has not been removed. It has been refined. It has been made more surgical. It has been made more efficient.
The moral weight of this lie is heavy. It falls on those who have been promised justice and received only a change in terminology. It falls on the families who have been broken by the enforcement of archaic laws. It falls on the communities that have been targeted for generations. The Reform Lie assumes that the public has forgotten the history of the struggle. It assumes that the public is satisfied with the crumbs of corporate legitimacy. It assumes that there is no understanding of the difference between the freedom to live and the permission to serve.
The narrative of the state must be rejected. The recognition must grow that every small step that leaves the core structure of the prohibition machine in place is a step away from justice. The government must be held accountable for the contradiction of its own law. The reality of the prohibition era must continue to be documented, to expose the lies that are told to justify the control, and to advocate for the total restoration of liberty. The struggle for the plant is not a struggle for a change in status. It is a struggle for the soul of the culture. It is a struggle to define what it means to be a free person in a society that seeks to regulate every choice. As NORML Deputy Director Paul Armentano noted regarding the order:
“Rescheduling fails to fully harmonize federal marijuana policy with the cannabis laws of many states, particularly the 24 states that have legalized its use and sale to adults.”
This is the core of the deception. The Reform Lie is the latest barrier to that freedom. It is a wall that must be dismantled, not by the government, but by the people who have lived the reality of the struggle.
The truth is simple, though the state works hard to obscure it. Cannabis is a part of the human experience. It has been used for healing, for creativity, for connection, and for joy for as long as historical records exist. The attempts by the state to control this relationship are an affront to human autonomy. They are based on fear, on ignorance, and on a desire for power. The reclassification to Schedule III is just the latest tactic in a long campaign to prevent people from fully embracing their own sovereignty. While the proponents of this move claim that:
“Today’s order marks a historical reversal in federal cannabis policy,”
It is a sign that the state is feeling the pressure, that it knows its position is untenable, but that it is not yet ready to concede.
A crossroads has been reached. Either the crumbs offered by the state are accepted, turning the public into participants in their own regulation, or the fight for the total and unconditional end of the prohibition machine continues. The Reform Lie can be accepted, or the truth can be demanded. The history of the culture is a history of resistance. It is a history of people who refused to be told what they could do, who they could be, or what they could consume. That history is the source of strength. It is the foundation upon which the future will be built. Permission from the state is not required to exist. Schedules, labels, and tax codes are not needed to define what is right. The truth is known, and it will continue to be shared until the last cage is empty and the prohibition machine is nothing but a memory.
The Reform Lie will continue to be told. The headlines will continue to scream about progress that does not exist. The state will continue to frame its maintenance of power as a move toward justice. But the deception will not hold. The patterns are visible. The history is known. The stakes are understood. The reality of the prohibition era will be documented, one article, one story, one voice at a time. This is not just a battle for a plant. It is a battle for the truth. And it is a battle that will be won, not because the state gives permission, but because the truth is on the side of the people. The prohibition machine is built on lies, and lies cannot stand forever against the weight of reality. The end of prohibition is coming, not through the actions of the state, but through the resolve of the people who have been fighting for it all along. The Reform Lie is the last gasp of a system that knows its time is over. We will not be fooled. We will not be silenced. We will be here, documenting the reality, telling the truth, and fighting for the culture until the day the plant is free.
©2026, Pot Culture Magazine. All rights reserved. This is the property of Pot Culture Magazine and is protected by U.S. and international copyright laws. Unauthorized reproduction, distribution, or transmission
of this work, in part or in whole, without the express written permission of Pot Culture Magazine, is strictly
prohibited.F O R T H E C U L T U R E B Y T H E C U L T U R E
The Digital Cage: Saint Lucia’s Traceability Trap
Saint Lucia has selected GrowerIQ as its national seed-to-sale traceability backbone, effectively finalizing a digital surveillance grid for its cannabis industry. By mandating enterprise software before establishing licensing frameworks, the government risks automating the exclusion of legacy farmers. This move trades cultural sovereignty for state-managed control, turning the cannabis industry into an extension of the…
by Pot Culture Magazine EditorsApril 23, 2026April 22, 2026Gov. Abigail Spanberger’s Virginia Sabotage
Virginia legalized possession, but Governor Abigail Spanberger sabotaged the retail market. By delaying sales until 2027 and gutting equity provisions, the Commonwealth institutionalized a half-legal trap. Consumers now navigate a system that treats possession as a right but supply as a crime, fueling an unchecked illicit market while abandoning promised reform. Spanberger’s public safety rhetoric…
by Pot Culture Magazine EditorsApril 21, 2026April 20, 20264/20 has been hollowed out by branding, corporate silence, and a culture that forgot its own history. While the industry sells holiday merch, Singapore executed a man for cannabis. The movement that once fought for autonomy now treats the plant like a commodity. This piece examines the cost of that betrayal and the culture left…
by Pot Culture Magazine EditorsApril 20, 2026April 24, 2026 #280E #AdministrativeLaw #cannabis #CannabisCommunity #CannabisCulture #CannabisCommunity #CarceralState #Culture #DEA #DepartmentOfJustice #DrugWar #FederalGovernment #Industry #Legalization #Marijuana #MarijuanaNews #NORML #Policy #PolicyFiction #PotCultureMagazine #Prohibition #Reform #ScheduleIII #StateSanctioned #Weed -
Using a Raspberry Pi 2 Model B as a router/firewall for the home LAN
Since 1999 I have been using a 1996 vintage DEC PII desktop as the router/firewall between the internet and my home network. The DEC computer came to me with Win95 (or possibly Win98) in 1998, got SuSE linux and started its mission as router and firewall (and CUPS server, and IMAP server, and various other server stuff). When upgrading the SuSE installation to a newer version went south, it spent a while running ThomasEz’s floppyfw, until I used a floppy net install to install debian potato, immediately switched it to debian testing, until debian woody arrived, when it was moved to debian stable, and then I just kept running “apt-get dist-upgrade” until I finally had it running debian 8 “jessie” on june 6 in 2015.
The old DEC desktop has survived its maker company, survived lightning strikes that have sent the power supplies and/or main boards of other computers on the same LAN into continously beeping mode (i.e. broken). However, in December 2015 it started acting up, and crashing with irregular intervals (sometimes two weeks, sometimes one day).
So… the time for a replacement would have to be not too far ahead. The question was what to replace it with?
The simplest solution would be to just get a wireless router with a cabled switch. But that would mean:
- No possibilities for SSH or mosh into the home LAN
- No ntop
- No support for netboot and TFTP in the home LAN
- Limited, cumbersome and inflexible firewall setup
My requirements were:
- Cheap
- Two wired NICs
- The ability to run debian
- Preferrably fanless
- Compact
ThomasEz immediately suggested using a raspberry pi with two NICs, but I thought that would be too puny, and I investigated alternatives like Shuttle Barebone DS57U but I found that the raspberry pi alternative was so cheap, I might as well order one.
And then it turned out to be so simple to set up so I had it up and running before I really had decided on anything, so now the r-pi is what I have.
This is what I ordered:
- Raspberry Pi 2 Model B Starter Kit
- TP-Link UE300 USB 3.0 to GbE Adapter (it was listed as being supported out of the box on raspberry pi)
Here’s what I did:
- Downloaded the Raspbian Jessie Lite image to a debian jessie computer and unpacked it into the /tmp directory
- Plugged an USB SD card reader into the debian computer, and followed the instructions in Installing operating system images on Linux
- I plugged the cheapest USB keyboard I could get from my local teknikmagasinet store into one of the USB port, yanked the HDMI cable from the DVD player and plugged the r-pi into the TV, plugged a network cable into the local LAN, and plugged in the power… and the raspberry pi booted quickly into the familiar debian login
- I logged in with the built-in “pi” user with password “raspberry”, and created my own user with the following command line command:
adduser sb
the changed the password of the root user and removed the pi user
- I copied in a public ssh keys from my other computers, and put them into the ~/.ssh/authorized_keys file and then opened /etc/ssh/sshd_conf in a text editor and modified it in the following way:
- Disabled root login by changing
PermitRootLogin without-password
to
PermitRootLogin no
- Disabled password login by changing
#PasswordAuthentication yes
to
PasswordAuthentication no
(removed the comment and changed “yes” to “no”)
- Disabled root login by changing
- Edited /etc/hostname to change the name from the default “raspberrypi” to “ocon”
- Rebooted the pi to check the startup state of the ssh daemon and ssh’d in
- Resized the disk to fill the entire SD card:
- Typed the command
raspi-config
- Selected
1 Expand Filesystem Ensures that all of the SD card storage is available to the OS
and got the response
Root partition has been resized.The filesystem will be enlarged upon the next reboot
- Rebooted the system to get the full 16GB in the file system
- Typed the command
- Updated the system by giving the following command line commands:
apt-get updateapt-get dist-upgrade
(the “update” command updates the local package database against the package servers. The “dist-upgrade” command upgrades all packages that have a newer version, and the required dependencies)
- Installed some useful software:
- GNU emacs (my favorite text editor)
apt-get install emacs
- mosh
apt-get install mosh
- git (I’ve got my home directory versioned in git)
apt-get install git
- rcs (I use it to version control operating system configuration files)
apt-get install rcs
- GNU emacs (my favorite text editor)
- I cloned my home directory in git and created a new branch (I have a different branch for each computer)
- I set the built-in NIC permanently as eth0:
export INTERFACE=eth0export MATCHADDR=`ip addr show $INTERFACE | grep ether | awk '{print $2}'`/lib/udev/write_net_rules - I added configuration for a second NIC by adding the following to /etc/network/interfaces:
# The internal network cardallow-hotplug eth1iface eth1 inet static address 10.10.10.1 netmask 255.255.255.0
- I plugged in the USB NIC to have it appear, and then made the USB NIC permanently eth1 with the following command line commands:
export INTERFACE=eth1export MATCHADDR=`ip addr show $INTERFACE | grep ether | awk '{print $2}'`/lib/udev/write_net_rules - Installed dnsmasq
apt-get install dnsmasq
- Edited /etc/dnsmasq.conf to make dnsmasq respond to DHCP requests on eth1:
- Removed the comment in front of
#interface=
and set “eth1” as the value:
interface=eth1
- Uncommented the domain directive
#domain=thekelleys.org.uk
and changed it to my domain
domain=hjemme.lan
- Uncommented the dhcp-range directive
#dhcp-range=192.168.0.50,192.168.0.150,12h
and changed it to a 10.10.10.* range with a 5h lease on the addresses
# Our HOME LAN 5h lease timedhcp-range=10.10.10.6,10.10.10.40,5h
- Removed the comment in front of
- Opened the /etc/hosts file in a text editor and added the raspberry pi itself, to so that DNS lookups of the raspberry pi will work in a LAN where the raspberry pi is handling the DHCP requests (dnsmasq will handle DNS requests for the IP addresses it has given DHCP leases to, as well as what it finds in the hosts file. The rest is delegated to the upstream DNS server)
127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopbackff02::1 ip6-allnodesff02::2 ip6-allrouters127.0.1.1 ocon# local hosts10.10.10.1 hjemme ocon hjemme.hjemme.lan ocon.hjemme.lan
- Edited the /etc/sysctl.conf file to set up IPv4 routing in the linux kernel, removed the comment in front of the net.ipv4.ip_forward line:
# Uncomment the next line to enable packet forwarding for IPv4net.ipv4.ip_forward=1
- ferm is a utility that makes it easy to set the routing and firewall rules at boot time
- Installed ferm using apt-get from a command line:
apt-get install ferm
- Modified the /etc/ferm/ferm.conf file to allow everything inside t oroute out, but only allow ssh in
@def $DEV_WORLD = eth0;@def $DEV_PRIVATE = eth1;def $NET_PRIVATE = 10.10.10.0/24;table filter { chain INPUT { policy DROP; # connection tracking mod state state INVALID DROP; mod state state (ESTABLISHED RELATED) ACCEPT; # allow local packet interface lo ACCEPT; # allow private net interface $DEV_PRIVATE ACCEPT; # respond to ping proto icmp ACCEPT; # allow IPsec proto udp dport 500 ACCEPT; proto (esp ah) ACCEPT; # allow SSH connections proto tcp dport ssh ACCEPT; } chain OUTPUT { policy ACCEPT; # connection tracking #mod state state INVALID DROP; mod state state (ESTABLISHED RELATED) ACCEPT; } chain FORWARD { policy DROP; # connection tracking mod state state INVALID DROP; mod state state (ESTABLISHED RELATED) ACCEPT; # connections from the internal net to the internet or # to other internal nets are allowed interface $DEV_PRIVATE ACCEPT; # the rest is dropped by the above policy }}table nat { chain POSTROUTING { # masquerade private IP addresses saddr $NET_PRIVATE outerface $DEV_WORLD MASQUERADE; }}
- Installed ferm using apt-get from a command line:
- The version of ferm in “jessie” doesn’t start at boot, because “jessie” dropped SYSV init in favour of systemd, and the version of ferm in “jessie” doesn’t have a systemd configuration, so I needed to manually download and install the version of ferm from debian testing (I downloaded from regular debian, since ferm doesn’t have anything platform specific):
cd /tmpwget http://ftp.no.debian.org/debian/pool/main/f/ferm/ferm_2.2-5_all.debdpkg --install /tmp/ferm_2.2-5_all.deb
- fail2ban monitors log files of daemons and adjust the firewall rules to temporary ban hosts it suspects of intrusion attempts. The debian (and raspbian) version of fail2ban will out of the box scan the logs for ssh intrusion attempts, so no configuration is necessary
- To have an easy way of monitoring the network traffic in and out of the home LAN, I installed ntop ng
apt-get install ntopng
after the installation it is possible to monitor the network traffic by accessing http://ocon.hjemme.lan:3000 (the interesting traffic will be seen after selecting eth1)
- The Network Time Protocol is how computers stay in sync, installing the ntp package will make the gateway keep network time, a
apt-get install ntp
- Opened the /etc/ntp.conf file in a text editor, and modified it to provide an NTP deamon for the home LAN, uncommented the “broadcast” line and modified the network match to match the 10.10.10.* network:
# If you want to provide time to your local subnet, change the next line.# (Again, the address is an example only.)broadcast 10.10.10.255
- Installed the apticron utility to make sure that the APT database is updated daily with new candidates for update
apt-get install apticron
The original plan was to run the raspberry pi headless, but since I had an old VGA only LCD display for the old DEC computer I might as well hook it up the raspberry pi, together with the cheap USB keyboard used for setup.
I bought an HDMI to VGA converter with the manufacturer id VLMP34900W0.20. I plugged it in between the display and the raspberry-pi the display stayed black. I edited the /boot/config.txt file, removing the comment in front of the hdmi_safe line:
# uncomment if you get no picture on HDMI for a default "safe" modehdmi_safe=1
I rebooted the raspberry pi, and this time the LCD displayed showed the boot messages as well as a normal console login prompt.
The raspberry pi 2 model B, with an extra USB NIC, a USB keyboard and connected to a VGA display using an HDMI to VGA converterAnd this is where the current state is. One initial concern was flash wear on the SD card, which doesn’t have the wear leveling features of a “real” SSD, so I had some plans on making the /var/log use tmpfs.
But I decided not to, since having real persistent logs is a useful thing for a gateway, and since 16GB is actually an awful lot of data if all you do is to write textual files. And ff the SD card wears out I’ll just by a new SD card, and make a new system. Since I now know how, this shouldn’t take long
#debian #dnsmasq #fail2ban #ferm #firewall #ipMasquerading #jessie #mosh #ntop #raspbian #raspbian8 #raspbianJessie #router #ssh