#awsglue — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #awsglue, aggregated by home.social.
-
When does #Iceberg beat #Parquet+projection on #AWSGlue, and when doesn't ?
An end-to-end #ETL PoC on #AWS to find out: producer, #Kinesis, two #Firehose paths, two #Glue jobs, #Athena.
🔮 Spoiler: how the data is read is the key to the choice.
In the article: every choice with its why, plus a few gems from some Glue experience 😄
-
When does #Iceberg beat #Parquet+projection on #AWSGlue, and when doesn't ?
An end-to-end #ETL PoC on #AWS to find out: producer, #Kinesis, two #Firehose paths, two #Glue jobs, #Athena.
🔮 Spoiler: how the data is read is the key to the choice.
In the article: every choice with its why, plus a few gems from some Glue experience 😄
-
AWS Glue now supports Microsoft Dynamics 365 as a data source
https://aws.amazon.com/about-aws/whats-new/2025/07/aws-glue-microsoft-dynamics-365-data-source/
This connector allows Glue users to build efficient ETL jobs that seamlessly connect to Microsoft Dynamics 365 as a data source.
#AWS #AwsGlue #Dynamics365 -
that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
-
that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
-
that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
-
that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
-
that feeling when you realizing you might be one of about six people in the world attempting to use an enterprise technology and it isn't clear that any of the six have ever gotten it to work.
(In this case an #AWSGlue to #Opensearch java library)
-
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product -
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product -
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product -
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product -
Why developer experience doesn't matter, as expressed by #AWSGlue and #Oracle
- Dev-ex is expensive, only developers enjoy it
- Punching the developer in the face (metaphorically) is immensely satisfying to product managers and is free
- If you don't metaphorically punch developers in the face they will get uppity and use your product.
- They don't deserve to use your product -
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
-
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
-
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
-
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
-
You thought you were done with Python3.9 and Python3.10? Nope, because you decided to skill up & try to use #AWSGlue
You think maybe at least the 2 flavors of glue (spark and ray) could at least both get to 3.10, but nope.
-
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.
-
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.
-
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.
-
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.
-
#AWSGlue doesn't use regular python packaging, it asks the user to create a zip of dependencies and the users own code. (I count 3 different ways & they're documented in English not code.)
So I started writing a #python packaging tool for that
#AWSLambda s kind of do the same thing, but I already named the package. Sigh.
-
@schizanon I understand the feeling but I am curious about what you hate specifically. Since I am into bioinformatics and data engineering, there are other services I find disturbingly buggy and badly designed and documented (like #awsglue and #awsOmics ) but always curious to know what other users experience. I am relatively new to #AWS but sometimes it surprises me how much time I invest helping #AWS debug software I am paying for. I understand small bugs but we have found major ones!
-
@schizanon I understand the feeling but I am curious about what you hate specifically. Since I am into bioinformatics and data engineering, there are other services I find disturbingly buggy and badly designed and documented (like #awsglue and #awsOmics ) but always curious to know what other users experience. I am relatively new to #AWS but sometimes it surprises me how much time I invest helping #AWS debug software I am paying for. I understand small bugs but we have found major ones!
-
@schizanon I understand the feeling but I am curious about what you hate specifically. Since I am into bioinformatics and data engineering, there are other services I find disturbingly buggy and badly designed and documented (like #awsglue and #awsOmics ) but always curious to know what other users experience. I am relatively new to #AWS but sometimes it surprises me how much time I invest helping #AWS debug software I am paying for. I understand small bugs but we have found major ones!
-
@schizanon I understand the feeling but I am curious about what you hate specifically. Since I am into bioinformatics and data engineering, there are other services I find disturbingly buggy and badly designed and documented (like #awsglue and #awsOmics ) but always curious to know what other users experience. I am relatively new to #AWS but sometimes it surprises me how much time I invest helping #AWS debug software I am paying for. I understand small bugs but we have found major ones!
-
@schizanon I understand the feeling but I am curious about what you hate specifically. Since I am into bioinformatics and data engineering, there are other services I find disturbingly buggy and badly designed and documented (like #awsglue and #awsOmics ) but always curious to know what other users experience. I am relatively new to #AWS but sometimes it surprises me how much time I invest helping #AWS debug software I am paying for. I understand small bugs but we have found major ones!
-
There is another quirk, not to say, an issue, in AWS Glue. In Python shell jobs, when using Python 3.9, pip is not aware of the pre-built analytics libraries when installing an external wheel. As a result, it resolves and downloads from the package registry even if the pre-built versions of pandas or numpy satisfy the wheel’s requirements.
#Python #AWS #AWSGlue #bug #pandas #numpy
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
-
There is another quirk, not to say, an issue, in AWS Glue. In Python shell jobs, when using Python 3.9, pip is not aware of the pre-built analytics libraries when installing an external wheel. As a result, it resolves and downloads from the package registry even if the pre-built versions of pandas or numpy satisfy the wheel’s requirements.
#Python #AWS #AWSGlue #bug #pandas #numpy
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
-
There is another quirk, not to say, an issue, in AWS Glue. In Python shell jobs, when using Python 3.9, pip is not aware of the pre-built analytics libraries when installing an external wheel. As a result, it resolves and downloads from the package registry even if the pre-built versions of pandas or numpy satisfy the wheel’s requirements.
#Python #AWS #AWSGlue #bug #pandas #numpy
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
-
There is another quirk, not to say, an issue, in AWS Glue. In Python shell jobs, when using Python 3.9, pip is not aware of the pre-built analytics libraries when installing an external wheel. As a result, it resolves and downloads from the package registry even if the pre-built versions of pandas or numpy satisfy the wheel’s requirements.
#Python #AWS #AWSGlue #bug #pandas #numpy
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
-
There is another quirk, not to say, an issue, in AWS Glue. In Python shell jobs, when using Python 3.9, pip is not aware of the pre-built analytics libraries when installing an external wheel. As a result, it resolves and downloads from the package registry even if the pre-built versions of pandas or numpy satisfy the wheel’s requirements.
#Python #AWS #AWSGlue #bug #pandas #numpy
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
-
There is an annoying issue in Spark 3.3.0 when AQE logical optimization rules can lead to invalid physical plans. This was fixed in Spark 3.4.0, and back ported to 3.3.1. However, AWS Glue v4 only has the Spark 3.3.0 environment available and there seems to be not plans to update it to 3.3.1 anytime soon. The only workaround is to disable AQE altogether, which, in turn, removes many performance benefits of Spark 3.
-
There is an annoying issue in Spark 3.3.0 when AQE logical optimization rules can lead to invalid physical plans. This was fixed in Spark 3.4.0, and back ported to 3.3.1. However, AWS Glue v4 only has the Spark 3.3.0 environment available and there seems to be not plans to update it to 3.3.1 anytime soon. The only workaround is to disable AQE altogether, which, in turn, removes many performance benefits of Spark 3.
-
There is an annoying issue in Spark 3.3.0 when AQE logical optimization rules can lead to invalid physical plans. This was fixed in Spark 3.4.0, and back ported to 3.3.1. However, AWS Glue v4 only has the Spark 3.3.0 environment available and there seems to be not plans to update it to 3.3.1 anytime soon. The only workaround is to disable AQE altogether, which, in turn, removes many performance benefits of Spark 3.
-
There is an annoying issue in Spark 3.3.0 when AQE logical optimization rules can lead to invalid physical plans. This was fixed in Spark 3.4.0, and back ported to 3.3.1. However, AWS Glue v4 only has the Spark 3.3.0 environment available and there seems to be not plans to update it to 3.3.1 anytime soon. The only workaround is to disable AQE altogether, which, in turn, removes many performance benefits of Spark 3.
-
There is an annoying issue in Spark 3.3.0 when AQE logical optimization rules can lead to invalid physical plans. This was fixed in Spark 3.4.0, and back ported to 3.3.1. However, AWS Glue v4 only has the Spark 3.3.0 environment available and there seems to be not plans to update it to 3.3.1 anytime soon. The only workaround is to disable AQE altogether, which, in turn, removes many performance benefits of Spark 3.
-
I attended the #AWSSummitLondon this week. I started with AWS ~6 months ago. It was gratifying to realise that I have learned a lot since then. I talked to a few experts and they told me I was in the right path and that the struggles I have with #AWSGlue are not only mine (they simply don’t support well #deltaLake ). My perfectionist self was relieved 😌 I didn’t solve any of my problems but sometimes it helps to realise you are not as stupid as your programming struggles make u feel sometimes… 😅
